arXiv:2501.16931cs.LGstat.AP2025-01被引 2

用分布视角评估模型性能,更真实反映训练不确定性。

Beyond Point Estimates: Distributional Uncertainty in Machine Learning Performance Evaluation

  • 将性能指标视为随机变量,通过分位数和置信区间分析其分布
  • 小样本(10-25次)下仍可有效进行统计推断,非参数置信区间适用
  • 适合对可靠性要求高的场景,如医疗、金融等关键应用

机器学习模型通常使用准确率、F1分数或均方误差等点估计来评估性能,但这类总结无法捕捉训练过程中的固有变异性,包括数据划分、初始化和超参数优化的随机性。本文提出从分布视角看待模型评估,将性能指标视为随机量而非固定值。不只关注平均值,而是通过分位数与置信区间分析性能指标的经验分布。研究基于真实数据的分类与回归任务案例,结合模拟实验验证点估计与区间估计方法。特别关注小样本情形(10-25次重复训练),符合实际中重复训练计算成本高的限制。结果表明,即使在小样本下,对性能分布的有意义统计推断依然可行,且标准非参数置信区间仍适用。该方法比均值评估更全面刻画变异性,支持风险导向的模型性能解读,适用于对可靠性要求高的应用场景。所提方法实现简单、通用性强,可作为标准评估流程的实用扩展。

原文摘要 · Abstract (English)

Machine learning models are often evaluated using point estimates of performance metrics such as accuracy, F1 score, or mean squared error. Such summaries fail to capture the inherent variability induced by stochastic elements of the training process, including data splitting, initialization, and hyperparameter optimization. This work proposes a distributional perspective on model evaluation by treating performance metrics as random quantities rather than fixed values. Instead of focusing solely on aggregate measures, empirical distributions of performance metrics are analyzed using quantiles and corresponding confidence intervals. The study investigates point and interval estimation of quantiles based on real-data use cases for classification and regression tasks, complemented by simulation studies for validation. Special emphasis is placed on small sample sizes, reflecting practical constraints in machine learning, where repeated training is computationally expensive. The results show that meaningful statistical inference on the underlying performance distribution is feasible even with sample sizes in the range of 10-25, while standard nonparametric confidence interval remain applicable under these conditions. The proposed approach provides a more detailed characterization of variability and uncertainty compared to mean-based evaluation and enables a more differentiated comparison of models. In particular, it supports a risk-oriented interpretation of model performance, which is relevant in applications where reliability is critical. The presented methods are easy to implement and broadly applicable, making them a practical extension to standard performance evaluation procedures in machine learning.

模型评估不确定性小样本分布建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。