arXiv:2502.12225cs.LGcs.AI2025-02被引 2

将标注者意见建模为不确定信念,提升主观任务的模型泛化能力

Subjective Logic Encodings

  • 用主观逻辑理论将标注视为观点,而非标准答案
  • 通过狄利克雷分布编码标注者的置信度、可靠性与分歧度
  • 适用于情感分析等主观任务,适合追求模型鲁棒性的研究者

现有学习方法通常假设存在金标准标签,将标注者分歧视为需消除的噪声。然而在情感分析、仇恨言论检测等主观任务中,分歧是自然存在的。为此,数据视角主义(data perspectivism)提出将标注视为标注者的观点,以保留任务固有的不确定性。现有方法仅利用分歧作为不确定性来源,而本文提出主观逻辑编码(SLEs),基于主观逻辑理论,将标签建模为狄利克雷分布,系统性地整合标注者置信度、可靠性和分歧度等多种不确定性信息。SLEs可统一多种标签编码方式,并提供基于分布匹配的目标函数来训练预测这些编码的模型。

原文摘要 · Abstract (English)

Many existing approaches for learning from labeled data assume the existence of gold-standard labels. According to these approaches, inter-annotator disagreement is seen as noise to be removed, either through refinement of annotation guidelines, label adjudication, or label filtering. However, annotator disagreement can rarely be totally eradicated, especially on more subjective tasks such as sentiment analysis or hate speech detection where disagreement is natural. Therefore, a new approach to learning from labeled data, called data perspectivism, seeks to leverage inter-annotator disagreement to learn models that stay true to the inherent uncertainty of the task by treating annotations as opinions of the annotators, rather than gold-standard facts. Despite this conceptual grounding, existing methods under data perspectivism are limited to using disagreement as the sole source of annotation uncertainty. To expand the possibilities of data perspectivism, we introduce Subjective Logic Encodings (SLEs), a flexible framework for constructing classification targets that explicitly encodes annotations as opinions of the annotators. Based on Subjective Logic Theory, SLEs encode labels as Dirichlet distributions and provide principled methods for encoding and aggregating various types of annotation uncertainty -- annotator confidence, reliability, and disagreement -- into the targets. We show that SLEs are a generalization of other types of label encodings as well as how to estimate models to predict SLEs using a distribution matching objective.

主观逻辑不确定性建模标注分歧

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。