arXiv:2603.24231cs.CLcs.SI2026-03

标注者一致但标签不一?问题出在多维立场被强行压缩成单一标签。

When Annotators Agree but Labels Disagree: The Projection Problem in Stance Detection

  • 提出‘投影问题’:多维立场压缩成单标签导致标注分歧。
  • 实验证明维度判断一致性高于标准标签,复杂目标差异更明显。
  • 适合做立场分析的模型改进与标注流程优化的研究者参考。

立场检测通常将文本分为支持、反对或中立三类,这一传统源自辩论分析,自SemEval-2016起被直接应用于社交媒体。然而,对复杂议题的态度并非单一维度。例如,有人认同气候科学却反对碳税,体现不同维度上的支持与反对。当标注者需将多维态度压缩为单一标签时,其侧重维度不同,造成分歧实为压缩策略差异而非理解混乱。我们通过跨五个议题(来自SemEval-2016、P-Stance、COVID-19-Stance)的标注研究发现,同一组标注者在每项议题上既给出标准立场标签,也评估特定维度上的立场,且维度分类数相同。在全部十五个目标-维度组合中,维度判断的一致性(AC1)普遍高于标签一致性;复杂议题如学校关闭的差距显著(AC1: 0.21 vs. 0.71),而单一人物如拜登的差距较小(AC1: 0.87 vs. 0.95)。

原文摘要 · Abstract (English)

Stance detection is nearly always formulated as classifying text into Favor, Against, or Neutral. This convention was inherited from debate analysis and has been applied without modification to social media since SemEval-2016. However, attitudes toward complex targets are not unitary. A person can accept climate science while opposing carbon taxes, expressing support on one dimension and opposition on another. When annotators must compress such multi-dimensional attitudes into a single label, different annotators may weight different dimensions, producing disagreement that reflects different compression choices rather than confusion. We call this the projection problem. We conduct an annotation study across five targets from three stance benchmarks (SemEval-2016, P-Stance, COVID-19-Stance), with the same three annotators labeling all targets. For each target, annotators assign both a standard stance label and per-dimension judgments along target-specific dimensions discovered through bottom-up analysis, using the same number of categories for both. Across all fifteen target--dimension pairs, dimensional agreement consistently exceeds label agreement. The gap appears to scale with target complexity: modest for a single-entity target like Joe Biden (AC1: 0.87 vs. 0.95), but large for a multi-faceted policy target like school closures (AC1: 0.21 vs. 0.71).

立场检测标注偏差多维分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。