Runtime evaluation of cognitive systems for non-deterministic multiple output classification problems

Aravind Kota Gopalakrishna (Corresponding author), Tanir Ozcelebi, Johan J. Lukkien, Antonio Liotta

Research output: Contribution to journalArticleAcademicpeer-review

2 Citations (Scopus)


Cognitive applications that involve complex decision making such as smart lighting have non-deterministic input–output relationships, i.e., more than one output may be acceptable for a given input. We refer them as non-deterministic multiple output classification (nDMOC) problems, which are particularly difficult for machine learning (ML) algorithms to predict outcomes accurately. Evaluating ML algorithms based on commonly used metrics such as Classification Accuracy (CA) is not appropriate. In a batch setting, Relevance Score (RS) was proposed as a better alternative, which determines how relevant a predicted output is to a given context. We introduce two variants of RS to evaluate ML algorithms in an online setting. Furthermore, we evaluate the algorithms using different metrics for two datasets that have non-deterministic input–output relationships. We show that instance-based learning provides superior RS performance and the RS performance keeps improving with an increase in the number of observed samples, even after the CA performance has converged to its maximum. This is a crucial result as it illustrates that RS is able to capture the performance of ML algorithms in the context of nDMOC problems while CA cannot.

Original languageEnglish
Pages (from-to)1005-1016
Number of pages12
JournalFuture Generation Computer Systems
Publication statusPublished - 1 Nov 2019


  • Classification problems
  • Cognitive systems
  • Human factors
  • Machine learning
  • Non-deterministic multiple output classification
  • Performance metric
  • Relevance score
  • Smart lighting


Dive into the research topics of 'Runtime evaluation of cognitive systems for non-deterministic multiple output classification problems'. Together they form a unique fingerprint.

Cite this