Evaluating Quadratic Weighted Kappa as the Standard Performance Metric for Automated Essay Scoring

Afrizal Doewes, Nughthoh Kurdhi, Akrati Saxena

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

384 Downloads (Pure)

Abstract

Automated Essay Scoring (AES) tools aim to improve the efficiency and consistency of essay scoring by using machine learning algorithms. In the existing research work on this topic, most researchers agree that human-automated score agreement remains the benchmark for assessing the accuracy of machine-generated scores. To measure the performance of AES models, the Quadratic Weighted Kappa (QWK) is commonly used as the evaluation metric. However, we have identified several limitations of using QWK as the sole metric for evaluating AES model performance. These limitations include its sensitivity to the rating scale, the potential for the so-called “kappa paradox” to occur, the impact of prevalence, the impact of the position of agreements in the diagonal agreement matrix, and its limitation in handling a large number of raters. Our findings suggest that relying solely on QWK as the evaluation metric for AES performance may not be sufficient. We further discuss insights into additional metrics to comprehensively evaluate the performance and accuracy of AES models.
Original languageEnglish
Title of host publicationProceedings of the 16th International Conference on Educational Data Mining
EditorsMingyu Feng, Tanja Käser, Partha Talukdar
PublisherInternational Educational Data Mining Society (IEDMS)
Pages103-113
Number of pages11
ISBN (Electronic)978-1-7336736-4-8
DOIs
Publication statusPublished - 11 Jul 2023
Event16th International Conference on Educational Data Mining, EDM 2023 - Bengaluru, India
Duration: 11 Jul 202314 Jul 2023

Conference

Conference16th International Conference on Educational Data Mining, EDM 2023
Abbreviated titleEDM 2023
Country/TerritoryIndia
CityBengaluru
Period11/07/2314/07/23

Fingerprint

Dive into the research topics of 'Evaluating Quadratic Weighted Kappa as the Standard Performance Metric for Automated Essay Scoring'. Together they form a unique fingerprint.

Cite this