Generating domain models from natural language text using NLP: a benchmark dataset and experimental comparison of tools

Fatma Bozyigit (Corresponding author), Tolgahan Bardakci, Alireza Khalilipour, Moharram Challenger, Guus Ramackers, Önder Babur, Michel R.V. Chaudron

Onderzoeksoutput: Bijdrage aan tijdschriftTijdschriftartikelAcademicpeer review

Samenvatting

Software requirements specification describes users’ needs and expectations on some target system. Requirements documents are typically represented by unstructured natural language text. Such texts are the basis for the various subsequent activities in software development, such as software analysis and design. As part of software analysis, domain models are made that describe the key concepts and relations between them. Since the analysis process is performed manually by business analysts, it is time-consuming and may introduce mistakes. Recently, researchers have worked toward automating the synthesis of domain models from textual software requirements. Current studies on this topic have limitations in terms of the volume and heterogeneity of experimental datasets. To remedy this, we provide a curated dataset of software requirements to be utilized as a benchmark by algorithms that transform textual requirements documents into domain models. We present a detailed evaluation of two text-to-model approaches: one based on a large-language model (ChatGPT) and one building on grammatical rules (txt2Model). Our evaluation reveals that both tools yield promising results with relatively high F-scores for modeling the classes, attributes, methods, and relationships, with txt2Model performing better than ChatGPT on average. Both tools have relatively lower performance and high variance when it comes to the relation types. We believe our dataset and experimental evaluation pave to way to advance the field of automated model generation from requirements.

Originele taal-2Engels
TijdschriftSoftware and Systems Modeling
VolumeXX
Nummer van het tijdschriftX
Vroegere onlinedatum8 mei 2024
DOI's
StatusE-publicatie vóór gedrukte publicatie - 8 mei 2024

Vingerafdruk

Duik in de onderzoeksthema's van 'Generating domain models from natural language text using NLP: a benchmark dataset and experimental comparison of tools'. Samen vormen ze een unieke vingerafdruk.

Citeer dit