Transformers and Meta-Tokenization in Sentiment Analysis for Software Engineering

Nathan W. Cassee (Corresponding author), Andrei E. Agaronian, Eleni Constantinou, Nicole Novielli, Alexander Serebrenik

Onderzoeksoutput: Bijdrage aan tijdschriftTijdschriftartikelAcademicpeer review

1 Citaat (Scopus)
29 Downloads (Pure)

Samenvatting

Sentiment analysis has been used to study aspects of software engineering, such as issue resolution, toxicity, and self-admitted technical debt. To address the peculiarities of software engineering texts, sentiment analysis tools often consider the specific technical lingo practitioners use. To further improve the application of sentiment analysis, there have been two recommendations: Using pre-trained transformer models to classify sentiment and replacing non-natural language elements with meta-tokens.
In this work, we benchmark five different sentiment analysis tools (two pre-trained transformer models and three machine learning tools) on 2 gold-standard sentiment analysis datasets. We find that pre-trained transformers outperform the best machine learning tool on only one of the two datasets, and that even on that dataset the performance difference is a few percentage points.
Therefore, we recommend that software engineering researchers should not just consider predictive performance when selecting a sentiment analysis tool because the best-performing sentiment analysis tools perform very similarly to each other (within 4 percentage points). Meanwhile, we find that meta-tokenization does not improve the predictive performance of sentiment analysis tools.
Both of our findings can be used by software engineering researchers who seek to apply sentiment analysis tools to software engineering data.
Originele taal-2Engels
Artikelnummer77
Aantal pagina's23
TijdschriftEmpirical Software Engineering
Volume29
Nummer van het tijdschrift4
Vroegere onlinedatum3 jun. 2024
DOI's
StatusGepubliceerd - jul. 2024

Vingerafdruk

Duik in de onderzoeksthema's van 'Transformers and Meta-Tokenization in Sentiment Analysis for Software Engineering'. Samen vormen ze een unieke vingerafdruk.

Citeer dit