Evaluating machine learning algorithms for applications with humans in the loop

A. Kota Gopalakrishna, T. Ozcelebi, J.J. Lukkien, A. Liotta

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

6 Citations (Scopus)
1 Downloads (Pure)


Applications employing data classification such as smart lighting that involve human factors such as perception lead to non-deterministic input-output relationships where more than one output may be acceptable for a given input. For these so called non-deterministic multiple output classification (nDMOC) problems, the relationship between the input and output may change over time making it difficult for the machine learning (ML) algorithms in a batch setting to make predictions for a given context. In this paper, we describe the nature of nDMOC problems and discuss the Relevance Score (RS) that is suitable in this context as a performance metric. RS determines the extent by which a predicted output is relevant to the user's context and behaviors, taking into account the inconsistencies that come with human (perception) factors. We tailor the RS metric so that it can be used to evaluate ML algorithms in an online setting at run-time. We assess the performance of a number of ML algorithms, using a smart lighting dataset with non-deterministic one-to-many input-output relationships. The results indicate that using RS instead of classification accuracy (CA) is suitable to analyze the performance of conventional ML algorithms applied to the category of nDMOC problems. Instance-based online ML gives the best RS performance. An interesting finding is that the RS keeps increasing with increasing number of samples, even after the CA performance converges.
Original languageEnglish
Title of host publicationProceedings of the IEEE 14th International Conference on Networking, Sensing and Control (ICNSC)
EditorsAntonio Guerrieri, Giancarlo Fortino, Athanasios V. Vasilakos, MengChu Zhou, Zofia Lukszo, Carlos Palau, Antonio Liotta, Andrea Vinci, Francesco Basile, Maria Pia Fanti
Place of PublicationPiscataway
PublisherInstitute of Electrical and Electronics Engineers
Number of pages6
ISBN (Electronic)978-1-5090-4429-0
ISBN (Print)978-1-5090-4430-6
Publication statusPublished - 1 Aug 2017
Event14th IEEE International Conference on Networking, Sensing and Control (ICNSC 2017) - Calabria, Italy
Duration: 16 May 201718 May 2017
Conference number: 14


Conference14th IEEE International Conference on Networking, Sensing and Control (ICNSC 2017)
Abbreviated titleICNSC 2017
Internet address


Dive into the research topics of 'Evaluating machine learning algorithms for applications with humans in the loop'. Together they form a unique fingerprint.

Cite this