Evaluating machine learning algorithms for applications with humans in the loop

Applications employing data classification such as smart lighting that involve human factors such as perception lead to non-deterministic input-output relationships where more than one output may be acceptable for a given input. For these so called non-deterministic multiple output classification (nDMOC) problems, the relationship between the input and output may change over time making it difficult for the machine learning (ML) algorithms in a batch setting to make predictions for a given context. In this paper, we describe the nature of nDMOC problems and discuss the Relevance Score (RS) that is suitable in this context as a performance metric. RS determines the extent by which a predicted output is relevant to the user's context and behaviors, taking into account the inconsistencies that come with human (perception) factors. We tailor the RS metric so that it can be used to evaluate ML algorithms in an online setting at run-time. We assess the performance of a number of ML algorithms, using a smart lighting dataset with non-deterministic one-to-many input-output relationships. The results indicate that using RS instead of classification accuracy (CA) is suitable to analyze the performance of conventional ML algorithms applied to the category of nDMOC problems. Instance-based online ML gives the best RS performance. An interesting finding is that the RS keeps increasing with increasing number of samples, even after the CA performance converges.