对比真人与合成软标签，发现前者更准且能稳定模型训练。
An Assessment of Human vs. Model Uncertainty in Soft-Label Learning and Calibration
- 用真实人类标注的软标签替代合成标签，分离不确定性与标签错误影响。
- 真人软标签使模型在难样本上校准更准，训练更稳定，准确率提升显著。
- 适合研究人机对不确定性的理解一致性或模型可解释性的人参考。
实现人类对齐的AI关键在于理解人类标注相较于合成标注的优势。尽管人类软标签通过捕捉不确定性提升了模型校准效果，但先前研究将此优势与隐含的误标数据修正（模式偏移）混淆，掩盖了软标签的真实作用。本文在MNIST及其合成变体上开展受控审计，重新标注子集以提取人类不确定性。通过解耦软标签监督与底层标签模式偏移，我们发现：虽然人类软标签带来准确率提升，其更大价值在于作为正则项，改善模型在困难样本上的校准表现，并促进训练过程的稳定性。数据地图分析显示，使用人类软标签训练的模型与人类不确定性一致，而合成标签训练的模型则无法对齐。本工作为人类-智能体不确定性对齐提供了一个诊断测试平台。
原文摘要 · Abstract (English)
Central to human-aligned AI is understanding the benefits of human-elicited labels over synthetic alternatives. While human soft-labels improve calibration by capturing uncertainty, prior studies conflate these benefits with the implicit correction of mislabeled data (mode shifts), obscuring true effects of soft-labels. We present a controlled audit of soft-label learning across MNIST and a synthetic variant, re-annotating subsets to extract human uncertainty. By decoupling soft-label supervision from underlying label mode shifts, we show that while human soft-labels do provide accuracy gains, their larger value lies in acting as a regularizer that improves model calibration on difficult samples and promotes stable convergence across training runs. Dataset cartography reveals models trained on human soft-labels mirror human uncertainty, whereas those trained on synthetic labels fail to align with humans. Broadly, this work provides a diagnostic testbed for human-AI uncertainty alignment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。