分析标注者与文本特征如何共同影响有害语言判断。
Who and What? Using Linguistic Features and Annotator Characteristics to Analyze Annotation Variation
- 融合标注者背景与文本语言特征，建模交互影响。
- 发现词汇线索和标注者态度显著影响判断结果。
- 不同数据集间效应差异大，需警惕结论泛化风险。
人类标注差异是自然语言处理中的核心现象：不同标注者对同一内容的判断存在视角差异，应被充分重视。数据收集实践因此转向增加标注者数量并发布细粒度标注数据，其中有害语言因主观性强而资源最丰富。尽管如此，关于‘谁’（标注者社会人口学特征、态度等）、‘什么’（文本语言属性）及其相互作用的研究仍较少。本文首次对四组有害语言检测基准数据集进行大规模分析，结合标注者特征、文本语言属性及其交互关系，构建统计驱动的综合图景。研究发现，交互作用至关重要，揭示了以往工作忽略的交叉效应；词汇线索与标注者态度起关键作用。然而，效应模式在不同数据集中差异显著，提示需谨慎对待模型泛化与迁移能力。
原文摘要 · Abstract (English)
Human label variation has been established as a central phenomenon in NLP: the perspectives different annotators have on the same item need to be embraced. Data collection practices thus shifted towards increasing the annotator numbers and releasing disaggregated datasets, harmful language being most resourced due to its high subjectivity. While this resulted in rich information about \textit{who} annotated (sociodemographics, attitudes, etc.), the \textit{what} (e.g., linguistic properties of items), and their interplay has received little attention. We present the first large-scale analysis of four reference datasets for harmful language detection, bringing together annotator characteristics, linguistic properties of the items, and their interactions in a statistically informed picture. We find that interactions are crucial, revealing intersectional effects ignored in previous work, and that a strong role is played by lexical cues and annotator attitudes. Effect patterns, however, vary considerably across datasets. This urges caution about generalization and transferability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。