arXiv:2501.19195cs.LGcs.AI2025-01被引 16

先优化分类精度,再校准置信度,提升模型预测可靠性

Rethinking Early Stopping: Refine, Then Calibrate

  • 分离校准与精炼误差,提出新视角指导训练
  • 训练只优化精炼误差,验证后用标准方法校准
  • 适用于各类分类器,显著提升预测准确性

机器学习分类器的概率预测在诸多领域中对准确和可解释决策至关重要。这类预测的质量通常通过交叉熵等合理损失函数评估,其分解为两个部分:校准误差衡量整体过/欠自信程度,精炼误差反映区分不同类别能力。本文提出一种新的变分形式来理解校准-精炼分解,揭示了后处理校准的新机制,并实现各成分的快速估计。基于此,我们提供理论与实证证据表明,训练过程中校准误差与精炼误差无法同时最小化。因此,仅依据验证损失选择最佳训练轮次,会得到两者均非最优的折中点。为此,我们提出在训练阶段仅最小化精炼误差(Refine),随后使用标准技术后处理校准校准误差(...then Calibrate)。该方法可无缝集成于任何分类器,且在多种分类任务中持续提升性能。

原文摘要 · Abstract (English)

Machine learning classifiers often produce probabilistic predictions that are critical for accurate and interpretable decision-making in various domains. The quality of these predictions is generally evaluated with proper losses, such as cross-entropy, which decompose into two components: calibration error assesses general under/overconfidence, while refinement error measures the ability to distinguish different classes. In this paper, we present a novel variational formulation of the calibration-refinement decomposition that sheds new light on post-hoc calibration, and enables rapid estimation of the different terms. Equipped with this new perspective, we provide theoretical and empirical evidence that calibration and refinement errors are not minimized simultaneously during training. Selecting the best epoch based on validation loss thus leads to a compromise point that is suboptimal for both terms. To address this, we propose minimizing refinement error only during training (Refine,...), before minimizing calibration error post hoc, using standard techniques (...then Calibrate). Our method integrates seamlessly with any classifier and consistently improves performance across diverse classification tasks.

模型校准分类任务概率预测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。