.png){kind=icon}
.png){kind=icon}
Return
Applied Fano’s inequality: a bound estimation on regression error with conditional entropy estimators
.png){kind=icon}
PRE
AI
Y
C
DOI：10.1007/s00521-026-12411-6.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Fano’s inequality provides a lower bound on the probability of error in any estimation task, which has been applied to the classification problem for Bayes error rate. However, for the regression problems with continuous response variables, the performance of the differential conditional entropy estimator limits the implementation of error bounds. This study proposes a new framework to address this gap, centered on two estimators: the Perturbed MArginal-Conditional Entropy (PMACE) estimator, which acts as a robust lower bound estimate of conditional entropy, and an enhanced Perturbed KNIFE (PKNIFE) estimator, which serves as a reliable upper bound. Together, they establish a reliable range for the conditional entropy. We validated this framework on 60 synthesized tasks. Experimental results show that our estimators are highly consistent: PMACE consistently functioned as a lower-bound estimator across all 60 tasks achieving the lowest average relative error of 0.189, while PKNIFE was the only estimator to consistently remain an upper-bound estimator. Moreover, we extend the Fano’s inequality to derive a novel bound on the coefficient of determination $$R^2$$ , thereby enhancing the interpretability. Across 16 real-world regression datasets, our framework successfully bounded the maximum achievable $$R^2$$ in 15 cases. The proposed estimators provide a practical tool for assessing the minimum achievable error in regression, and the approach can be adapted to classification problems as well.
Keywords:
Minimum achievable error
Predictability analysis
Conditional entropy estimation
Information theory
Regression problem
Journal
IF:
4.5
Papers:
729
Citations:
3.2W
Organization
Cited Papers