arXiv:2506.20209cs.CLcs.AI2025-06IJCAI被引 6

用软标签捕捉人类分歧,让NLP模型更包容多元观点。

Perspectives in Play: A Multi-Perspective Approach for More Inclusive NLP Systems

  • 用软标签代替单一真值,保留标注者个体差异
  • 在仇恨言论等任务上F1得分更高,更贴近人类分布
  • 适合关注公平性与主观性任务的研究者

在自然语言处理中,传统方法通常通过聚合标注者意见来确定单一真值,但忽略了个体差异可能带来的少数观点被边缘化问题,尤其在主观任务中。本文提出一种多视角方法,采用软标签建模不同标注者的立场,以促进更具包容性和多元性的下一代模型发展。我们在仇恨言论、讽刺、攻击性语言和立场检测等多种主观分类任务中进行了广泛分析,结果表明:该方法不仅在Jensen-Shannon散度(JSD)上更接近人类标签分布,还在分类性能上优于传统方法(更高F1分数)。然而,在讽刺和立场检测等高度主观任务中,模型置信度较低,反映其内在不确定性。借助可解释AI(XAI),我们进一步分析了模型预测中的不确定性,揭示了有意义的洞察。

原文摘要 · Abstract (English)

In the realm of Natural Language Processing (NLP), common approaches for handling human disagreement consist of aggregating annotators' viewpoints to establish a single ground truth. However, prior studies show that disregarding individual opinions can lead can lead to the side effect of underrepresenting minority perspectives, especially in subjective tasks, where annotators may systematically disagree because of their preferences. Recognizing that labels reflect the diverse backgrounds, life experiences, and values of individuals, this study proposes a new multi-perspective approach using soft labels to encourage the development of the next generation of perspective aware models, more inclusive and pluralistic. We conduct an extensive analysis across diverse subjective text classification tasks, including hate speech, irony, abusive language, and stance detection, to highlight the importance of capturing human disagreements, often overlooked by traditional aggregation methods. Results show that the multi-perspective approach not only better approximates human label distributions, as measured by Jensen-Shannon Divergence (JSD), but also achieves superior classification performance (higher F1 scores), outperforming traditional approaches. However, our approach exhibits lower confidence in tasks like irony and stance detection, likely due to the inherent subjectivity present in the texts. Lastly, leveraging Explainable AI (XAI), we explore model uncertainty and uncover meaningful insights into model predictions.

多视角软标签包容性主观任务

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。