不同评估指标所需标注人数不同，用软标签能更好捕捉争议性数据。
Metric-Dependent Annotation Saturation for Learning from Label Distributions
- 根据评估指标差异调整标注人数：熵相关需20-50人，分布匹配仅需10人
- 软标签使模型识别分歧能力提升至r=0.643，远超标签平滑的r=0.45-0.49
- 适用于关注标注效率与争议样本识别的研究者
当标注者对标签存在分歧时，这种分歧本身携带有效信息——而捕获该信息所需的标注人数取决于评估指标。我们在ChaosNLI数据集（每项有100个独立标注）上微调NLI模型，发现指标依赖的标注饱和现象：在三分类NLI任务中，衡量模型识别分歧能力的熵相关需约20-50名标注者才能收敛，而衡量分布匹配度的KL散度在约10名标注者时即达到87%-95%的性能提升。这一发现基于关键观察：软标签包含项目级信号，标签平滑无法复现。在五种平滑强度下，熵相关保持在r≈0.45-0.49，而软标签可达r=0.643（p<0.001）；逐项分析表明，平滑无法区分模糊项与明确项。该优势在DeBERTa、RoBERTa及非NLI预训练基线模型中均复现，并在内容安全跨域评估中初步验证。结果表明，标注预算应依据目标评估指标动态设定，而非统一分配。
原文摘要 · Abstract (English)
When annotators disagree on a label, the disagreement itself carries signal -- and the number of annotators needed to capture it depends on the evaluation metric. We fine-tune NLI models on label distributions subsampled from ChaosNLI, a dataset providing 100 independent annotator judgments per item, and identify metric-dependent saturation. In our 3-class NLI setting, entropy correlation -- whether the model identifies which items elicit disagreement -- requires N ~ 20-50 annotators to converge, while distributional match (KL divergence) saturates by N ~ 10 (87-95% of improvement across five model seeds). This finding rests on a prior observation: soft labels carry item-specific signal that label smoothing cannot replicate. Across five smoothing intensities, entropy correlation clusters at r ~ 0.45-0.49, while soft labels reach r = 0.643 (p < 0.001); per-item analysis traces this gap to smoothing's inability to distinguish ambiguous items from clear ones. The soft-label advantage replicates across two architectures (DeBERTa, RoBERTa), a non-NLI-pretrained baseline, and an exploratory cross-domain evaluation on content safety. These results suggest that annotation budgets should be informed by the target evaluation metric rather than set uniformly.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。