提出在复杂数据下准确估算贝叶斯误差率的方法,kNN表现最优。
Bayes Error Rate Estimation in Difficult Situations
- 采用kNN方法估算贝叶斯误差率,适用于多变量未知分布场景。
- 每类需1000样本才能使95%置信区间低于5%,4特征时需2500样本。
- 适合评估模型性能上限,尤其对小样本或高维数据问题有参考价值。
贝叶斯误差率(BER)是由于数据内在不确定性导致的任何机器学习模型可实现泛化分类精度的根本极限。BER估计器能揭示分类问题的难度,并设定最优分类性能预期。为确保估计器具备实用性,必须在少量样本下对多元问题且类别分布未知的情况下保持准确性。通过蒙特卡洛模拟使用合成数据,深入评估其在二分类中的准确性并获得置信区间。为检验实际应用可行性,引入新的非线性多模态测试场景,每个场景运行2500次蒙特卡洛模拟,覆盖广泛BER值。比较k近邻(kNN)、广义Henze-Penrose(GHP)散度和核密度估计(KDE)方法后发现,kNN在非参数估计中显著更准确。要达到95%置信区间小于5%的目标,每类最少需1000个样本;特征增加时样本需求上升,4个特征时每类需2500样本。尽管其他估计器在特征增多时精度提升,但仍无法满足目标区间要求。
原文摘要 · Abstract (English)
The Bayes Error Rate (BER) is the fundamental limit on the achievable generalizable classification accuracy of any machine learning model due to inherent uncertainty within the data. BER estimators offer insight into the difficulty of any classification problem and set expectations for optimal classification performance. In order to be useful, the estimators must also be accurate with a limited number of samples on multivariate problems with unknown class distributions. To determine which estimators meet the minimum requirements for "usefulness", an in-depth examination of their accuracy is conducted using Monte Carlo simulations with synthetic data in order to obtain their confidence bounds for binary classification. To examine the usability of the estimators for real-world applications, new non-linear multi-modal test scenarios are introduced. In each scenario, 2500 Monte Carlo simulations per scenario are run over a wide range of BER values. In a comparison of k-Nearest Neighbor (kNN), Generalized Henze-Penrose (GHP) divergence and Kernel Density Estimation (KDE) techniques, results show that kNN is overwhelmingly the more accurate non-parametric estimator. In order to reach the target of an under 5% range for the 95% confidence bounds, the minimum number of required samples per class is 1000. As more features are added, more samples are needed, so that 2500 samples per class are required at only 4 features. Other estimators do become more accurate than kNN as more features are added, but continuously fail to meet the target range.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。