Relevance in cyber-physical systems with humans in the loop

In cyber-physical systems such as intelligent lighting, the system responds autonomously to observed changes in the environment. In such systems, more than one output may be acceptable for a given input scenario. This type of relationship between the input and output makes it difficult to analyze machine learning algorithms using commonly used performance metrics such as classification accuracy (CA). CA only measures whether a predicted output is right or not, whereas it is more important to determine whether the predicted output is relevant for the given context or not. In this direction, we introduce a new metric, the relevance score (RS) that is effective for the class of applications where user perception leads to non-deterministic input-output relationships. RS determines the extent by which a predicted output is relevant to the user's context and behaviors, taking into account the variability and bias that come with human perception factors. We assess the performance of a number of machine learning algorithms, using different datasets, including data from an intelligent lighting pilot. We find that using RS instead of CA is appropriate to analyze the performance of conventional machine learning algorithms, particularly for the class of non-deterministic multiple-output problems. Our method may be applied to other scenarios in which cyber-physical systems involve humans in the control loop.