Evaluating Classification Performance: Receiver Operating Characteristic and Expected Utility

Yueran Yang

doi:10.1037/met0000515

Evaluating Classification Performance: Receiver Operating Characteristic and Expected Utility

2024-10-01

3

OA

AI

Y

Yueran Yang *

DOI：10.1037/met0000515

原文链接

原文求助

摘要

En 中文

Transitional Abstract How should humans and machines improve classification performance? Past use of receiver operating characteristic (ROC) analysis has focused on improving classification performance by improving a classifier's accuracy, which is indicated by the area under the classifier's ROC curve. This article, however, shows that ROC analysis inherently relates to expected utility analysis and the objective underlying ROC analysis is to maximize expected utility of classification. This article develops a tool to visualize a classifier's expected utility in its ROC space-expected utility lines (EU lines). Analyzing ROC curves with EU lines, this article shows how ROC analysis could be used to evaluate and compare classifiers' expected utilities for any given prior probabilities and utilities of classification outcomes. The analysis reveals that expected utility depends not only on a classifier's accuracy, but more importantly on the classifier's operating point. Therefore, ROC analysis should be used not only to choose an optimal classifier, but more importantly to locate its optimal operating point to maximize expected utility. Inspired by parameters involved in estimating expected utility, this article goes beyond ROC analysis and discusses other possible ways to increase expected utility. In addition to building important theoretical foundations for properly understanding and using ROC and expected utility analyses, this article sheds light on how researchers and practitioners can improve classification performance for policy or decision processes that involve binary classifications. One primary advantage of receiver operating characteristic (ROC) analysis is considered to be its ability to quantify classification performance independently of factors such as prior probabilities and utilities of classification outcomes. This article argues the opposite. When evaluating classification performance, ROC analysis should consider prior probabilities and utilities. By developing expected utility lines (EU lines), this article shows the connection between a classifier's ROC curve and expected utility of classification. In particular, EU lines can be used to estimate expected utilities when classifiers operate at any ROC point for any given prior probabilities and utilities. EU lines are useful across all situations-no matter if one examines a single classifier or compares multiple classifiers, if one compares classifiers' potential to maximize expected utilities or classifiers' actual expected utilities, and if the ROC curves are full or partial, continuous or discrete. The connection between ROC and expected utility analyses reveals the common objective underlying these two methods: to maximize expected utility of classification. Particularly, ROC analysis is useful in choosing an optimal classifier and its optimal operating point to maximize expected utility. Yet, choosing a classifier and its operating point (i.e., changing conditional probabilities) is not the only way to increase expected utility. Inspired by parameters involved in estimating expected utility, this article also discusses other approaches to increase expected utility beyond ROC analysis.

Keyword:

classification

receiver operating characteristic (ROC) analysis

expected utility

area under an ROC curve (AUC)

signal detection theory

Evaluating Classification Performance: Receiver Operating Characteristic and Expected Utility

摘要

期刊

机构

引用论文