arXiv:2502.12581stat.MLcs.AI2025-02被引 4

揭示多数投票在何种噪声下最优,为数据标注提供理论依据

The Majority Vote Paradigm Shift: When Popular Meets Optimal

  • 分析多数投票在特定噪声和类别分布下的最优性条件
  • 证明当标注噪声低于阈值时,多数投票可达到理论最低误差
  • 适合需要高效可靠标签聚合的研究者与数据标注团队

可靠的数据标注通常需多个标注者参与。然而人类标注存在误差,因此常通过聚合多标注结果来提升判断置信度。其中,简单且广泛使用的多数投票(MV)选择得票最多的类别作为最终标签。尽管重要,但MV的最优性尚未被深入研究。本文首次刻画了在何种条件下MV能达到标签估计误差的理论下限。结果表明,在给定类别分布下,当标注噪声低于某一容忍限度时,MV可最优恢复真实标签。该最优性证明为标签聚合方法选择提供了更严谨的依据,替代了耗时耗力且仍受人为不确定性影响的专家标注、黄金标准等传统做法。在合成数据与真实数据上的实验验证了理论结论。

原文摘要 · Abstract (English)

Reliably labelling data typically requires annotations from multiple human workers. However, humans are far from being perfect. Hence, it is a common practice to aggregate labels gathered from multiple annotators to make a more confident estimate of the true label. Among many aggregation methods, the simple and well known Majority Vote (MV) selects the class label polling the highest number of votes. However, despite its importance, the optimality of MV's label aggregation has not been extensively studied. We address this gap in our work by characterising the conditions under which MV achieves the theoretically optimal lower bound on label estimation error. Our results capture the tolerable limits on annotation noise under which MV can optimally recover labels for a given class distribution. This certificate of optimality provides a more principled approach to model selection for label aggregation as an alternative to otherwise inefficient practices that sometimes include higher experts, gold labels, etc., that are all marred by the same human uncertainty despite huge time and monetary costs. Experiments on both synthetic and real world data corroborate our theoretical findings.

数据标注多数投票理论分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。