用硬标签训练比软标签更优,尤其在标注稀疏时。
Same Target, Different Basins: Hard vs. Soft Labels for Annotator Distributions

- 采用多轮循环或随机采样硬标签,替代传统软标签方法。
- 标注少时硬标签提升显著,且偏差越大效果越明显。
- 适合标注不一致场景,尤其原始投票数据可用时。
当标注者意见不一时,这种分歧可能反映认知不确定性而非简单噪声。本文研究将投票结果转化为硬标签的策略,作为合并投票或直接使用经验软标签的替代方案。重点比较两种硬标签方法:多轮循环(multipass)保持数据集规模不变,逐轮遍历观测到的投票;随机标签采样(SLS)在每轮初始随机选取一个标签。在CIFAR-10H上,当每个样本仅有少量标注时,硬标签方法优于软标签训练,且当稀疏经验目标与完整标注分布距离越远,提升越显著。当完整标注分布可用时,两种硬标签方法性能与软标签相当。通过确定性对照和打乱SLS作为控制实验验证有效性。还证明SLS与软标签交叉熵优化同一期望目标。硬标签训练收敛至更平坦的损失盆地,支持证据来自对SVHN和CIFAR-100的分布外检测表现。总体表明,当有原始投票计数时,multipass是强实践默认选择;SLS则为轻量级替代方案,在少数标注时仍具竞争力,全标注下可媲美软标签训练。
原文摘要 · Abstract (English)
When annotators disagree, that disagreement can reflect epistemic uncertainty rather than simple label noise. We study hard-label delivery as an alternative to the usual choices of collapsing votes to a single label or training directly on the empirical soft-label distribution. We focus on two primary hard-label methods: multipass, which cycles through observed votes while keeping the dataset size fixed, and stochastic label sampling (SLS), which samples one label per example at the start of each epoch. On CIFAR-10H, we find that when only a small number of annotations per example is available, hard-label delivery improves over soft-label training, with larger improvements where the sparse empirical target is farther from the full annotator distribution. When full annotator distributions are available, both hard-label methods match soft-label training. We use deterministic control as an ablation of multipass and shuffled SLS as a control that breaks the example-to-distribution match. We also show that SLS and soft-label cross-entropy optimize the same expected objective. Hard-label delivery also converges to flatter basins, with supporting descriptive evidence from OOD detection on SVHN and CIFAR-100. Overall, these results suggest that multipass is a strong practical default when raw vote counts are available, while SLS offers a lightweight alternative that remains competitive when only a few votes per example are available and matches soft-label training when full annotator distributions are available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。