Samenvatting
Cognitive applications that involve complex decision making such as smart lighting have non-deterministic input–output relationships, i.e., more than one output may be acceptable for a given input. We refer them as non-deterministic multiple output classification (nDMOC) problems, which are particularly difficult for machine learning (ML) algorithms to predict outcomes accurately. Evaluating ML algorithms based on commonly used metrics such as Classification Accuracy (CA) is not appropriate. In a batch setting, Relevance Score (RS) was proposed as a better alternative, which determines how relevant a predicted output is to a given context. We introduce two variants of RS to evaluate ML algorithms in an online setting. Furthermore, we evaluate the algorithms using different metrics for two datasets that have non-deterministic input–output relationships. We show that instance-based learning provides superior RS performance and the RS performance keeps improving with an increase in the number of observed samples, even after the CA performance has converged to its maximum. This is a crucial result as it illustrates that RS is able to capture the performance of ML algorithms in the context of nDMOC problems while CA cannot.
| Originele taal-2 | Engels |
|---|---|
| Pagina's (van-tot) | 1005-1016 |
| Aantal pagina's | 12 |
| Tijdschrift | Future Generation Computer Systems |
| Volume | 100 |
| DOI's | |
| Status | Gepubliceerd - 1 nov. 2019 |
Financiering
This work was supported by Smart Context-aware Services (SmaCS), The Netherlands project through the Point One grant no. 10012172 . The authors would like to thank TU/e Intelligent Lighting Institute for providing the Breakout Dataset. Aravind Kota Gopalakrishna was a Ph.D. Student at the System Architecture and Networking (SAN) Group, Department of Mathematics and Computer Science, Eindhoven University of Technology, The Netherlands. Before joining SAN, he was associated with the TCS Innovation Labs, Bangalore, India as a Researcher from Dec 2008–Dec 2010 working on Routing issues in Wireless Sensor Networks for specific applications, Building Network Architectures, etc. He has completed Masters in Network Engineering from Manipal University, India and Bachelors in Electronics & Communication Engineering from Vishvesvaraya Technological University, India. His research interests include cognitive systems, context-aware/intelligent systems, applied machine learning, data mining and statistical analysis. Tanir Ozcelebi is an Assistant Professor in the Department of Mathematics and Computer Science at Eindhoven University of Technology (TU/e). He received his PhD in Electrical Engineering from Koc University, Istanbul in 2006. Then, he joined the System Architecture and Networking Research (SAN) group at TU/e as a postdoctoral researcher. Since 2013, Tanir has been research program manager for the Bright Environments research program of TU/e Intelligent Lighting Institute. Also, he is a core team member of the Internet of Things research program of the Data Science Center Eindhoven (DSC/e), Chairman of the Educational Committee Bachelor Computer Science and member of the TU/e wide Joint Program Committee. His main research interests are life-cycle management for resource-constrained embedded devices, architecture development for smart spaces and Internet of Things, as well as resource and QoS management and data analytics for networked services. Johan J. Lukkien is a Full Professor in the section Security and Embedded Networked Systems (SENS) at the Department of Mathematics and Computer Science, Eindhoven University of Technology (TU/e). Since 2002, he has been the Chair of the System Architecture and Networking Research within SENS. His areas of expertise include probability theory, statistics, software, algorithms, control systems, resource constrained systems, real-time systems and system architecture. Since 2000, his research focus has shifted to the application domain of networked resource-constrained embedded systems with a focus on architecture and protocols. He has contributed mainly in the area of component-based middleware for resource-constrained devices, distributed coordination, Quality of Service in networked systems and schedulability analysis in real-time systems. Antonio Liotta is a Professor of Data Science and the founding director of the Data Science Centre, University of Derby, UK. He is the director of the Joint Intellisensing Lab (with nodes in the UK, Netherlands, Italy, Australia and China); and a Guest Professor at Shanghai Ocean University, China and at Eindhoven University of Technology, NL. His team is at the forefront of research in network and data science, specifically in the context of Smart Cities, Internet of Things, and smart sensing. He studies the complexity of modern systems from the viewpoints of complex networks, machine learning, and artificial intelligence. He is the Editor-in-Chief of the Springer Internet of Things book series and associate editor of the Elsevier Information Fusion Journal.