研究标签聚合如何影响少数意见，揭示数据偏见与模型训练的潜在风险
Exploring the Influence of Label Aggregation on Minority Voices: Implications for Dataset Bias and Model Training
- 对比主流标签聚合方法对少数意见的压制效应
- 发现少数意见虽小但具价值，影响真实标签分布
- 适合关注数据公平性与模型偏见的研究者
人工标注中的分歧通常通过剔除不可靠标注者并采用多数投票或专家判断等标签聚合策略来解决。然而，这类方法可能无意中压制或低估少数但同样合理的观点。本文研究标准标签聚合策略在性别歧视检测任务中对少数意见代表性的影响，分析少数标注的质量与价值，并考察其对最终黄金标签类别分布的影响，以及由此带来的模型训练行为差异。最后讨论各方法引入的潜在偏差及其在模型中被放大的机制。
原文摘要 · Abstract (English)
Resolving disagreement in manual annotation typically consists of removing unreliable annotators and using a label aggregation strategy such as majority vote or expert opinion to resolve disagreement. These may have the side-effect of silencing or under-representing minority but equally valid opinions. In this paper, we study the impact of standard label aggregation strategies on minority opinion representation in sexism detection. We investigate the quality and value of minority annotations, and then examine their effect on the class distributions in gold labels, as well as how this affects the behaviour of models trained on the resulting datasets. Finally, we discuss the potential biases introduced by each method and how they can be amplified by the models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。