.png){kind=icon}
.png){kind=icon}
Return
Runtime evaluation of cognitive systems for non-deterministic multiple output classification problems
.png){kind=icon}
OA
AI
DOI：10.1016/j.future.2019.05.043.png)
.png){kind=icon}
.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Cognitive applications that involve complex decision making such as smart lighting have non-deterministic input-output relationships, i.e., more than one output may be acceptable for a given input. We refer them as non-deterministic multiple output classification (nDMOC) problems, which are particularly difficult for machine learning (ML) algorithms to predict outcomes accurately. Evaluating ML algorithms based on commonly used metrics such as Classification Accuracy (CA) is not appropriate. In a batch setting, Relevance Score (RS) was proposed as a better alternative, which determines how relevant a predicted output is to a given context. We introduce two variants of RS to evaluate ML algorithms in an online setting. Furthermore, we evaluate the algorithms using different metrics for two datasets that have non-deterministic input-output relationships. We show that instance-based learning provides superior RS performance and the RS performance keeps improving with an increase in the number of observed samples, even after the CA performance has converged to its maximum. This is a crucial result as it illustrates that RS is able to capture the performance of ML algorithms in the context of nDMOC problems while CA cannot. (C) 2019 Elsevier B.V. All rights reserved.
Keywords:
Cognitive systems
Machine learning
Classification problems
Performance metric
Relevance score
Non-deterministic multiple output classification
Human factors
Smart lighting
AI Summary
Key information extracted from the uploaded paper, including a brief overview, abstract, background, key highlights, visual analysis, and future outlook.
Journal
F
IF:
6.1
Papers:
6.8K
Citations:
2.3W