Samenvatting
Automated Essay Scoring (AES) tools aim to improve the efficiency and consistency of essay scoring by using machine learning algorithms. In the existing research work on this topic, most researchers agree that human-automated score agreement remains the benchmark for assessing the accuracy of machine-generated scores. To measure the performance of AES models, the Quadratic Weighted Kappa (QWK) is commonly used as the evaluation metric. However, we have identified several limitations of using QWK as the sole metric for evaluating AES model performance. These limitations include its sensitivity to the rating scale, the potential for the so-called “kappa paradox” to occur, the impact of prevalence, the impact of the position of agreements in the diagonal agreement matrix, and its limitation in handling a large number of raters. Our findings suggest that relying solely on QWK as the evaluation metric for AES performance may not be sufficient. We further discuss insights into additional metrics to comprehensively evaluate the performance and accuracy of AES models.
Originele taal-2 | Engels |
---|---|
Titel | Proceedings of the 16th International Conference on Educational Data Mining |
Redacteuren | Mingyu Feng, Tanja Käser, Partha Talukdar |
Uitgeverij | International Educational Data Mining Society (IEDMS) |
Pagina's | 103-113 |
Aantal pagina's | 11 |
ISBN van elektronische versie | 978-1-7336736-4-8 |
DOI's | |
Status | Gepubliceerd - 11 jul. 2023 |
Evenement | 16th International Conference on Educational Data Mining, EDM 2023 - Bengaluru, India Duur: 11 jul. 2023 → 14 jul. 2023 |
Congres
Congres | 16th International Conference on Educational Data Mining, EDM 2023 |
---|---|
Verkorte titel | EDM 2023 |
Land/Regio | India |
Stad | Bengaluru |
Periode | 11/07/23 → 14/07/23 |