Optimal ROC-Based Classification and Performance Analysis under Bayesian Uncertainty Models

Lori A. Dalton

doi:10.1109/TCBB.2015.2465966

Return

Optimal ROC-Based Classification and Performance Analysis under Bayesian Uncertainty Models

2016-07-01

PRE

Lori A. Dalton *

DOI：10.1109/TCBB.2015.2465966

Original

Original request for help

Save

Abstract

En 中文

Popular tools to evaluate classifier performance are the false positive rate (FPR), true positive rate (TPR), receiver operator characteristic (ROC) curve, and area under the curve (AUC). Typically, these quantities are estimated from training data using simple resampling and counting methods, which have been shown to perform poorly when the sample size is small, as is typical in many applications. This work takes a model-based approach in classifier training and performance analysis, where we assume the true population densities are members of an uncertainty class of distributions. Given a prior over the uncertainty class and data, we form a posterior and derive optimal mean-squared-error (MSE) FPR and TPR estimators, as well as the sample-conditioned MSE performance of these estimators. The theory also naturally leads to optimal ROC and AUC estimators. Finally, we develop a Neyman-Pearson-based approach to optimal classifier design, which maximizes the estimated TPR for a given estimated FPR. These tools are optimal over the uncertainty class of distributions given the sample, and are available in closed form or can be easily approximated for many models. Applications are demonstrated on both synthetic and real genomic data. MATLAB code and simulations results are available in the online supplementary material.

Keywords:

Classification

error estimation

Bayesian estimation

receiver operating characteristic

area under the curve

AI Summary

Key information extracted from the uploaded paper, including a brief overview, abstract, background, key highlights, visual analysis, and future outlook.