不同定义导致媒体偏见标注差异，揭示了标注规范的关键问题。
Definitional Sensitivity in Media Bias Detection: A Multi-Definition Dataset and Benchmark
- 用多定义对比实验检验概念框架对偏见标注的影响
- 人类与大模型在不同定义下标注差异显著，尤其大模型更敏感
- 适合研究标注规范、提示工程及偏见检测可复现性的学者
媒体偏见检测依赖于定义和示例来界定何为偏见，但这些定义在不同数据集间常有差异，甚至相同名称下也隐含不同含义。这种差异使训练于同一偏见类别模型是否学习到相同概念变得模糊，而这一问题长期被忽视。我们通过354名参与者开展的组间实验，以及四类大语言模型的平行评估，考察了定义选择对偏见标注的影响。参与者和模型对六篇新闻文章在四个偏见类别下进行评分，使用概念框架和详尽程度不同的定义。在8,496次人类与28,800次大模型评分中发现：概念框架显著影响标注结果，且对大模型的影响更强；而保持构念一致的细节扩展则无明显影响。研究讨论了标注协议与提示测量中的构念明确性意义，并指出定义敏感性可能蔓延至下游分类任务。同时发布MUDD——多定义偏见检测数据集。
原文摘要 · Abstract (English)
Media bias detection relies on definitions and examples that specify what counts as bias, yet these specifications often vary across datasets or remain implicit, even when given the same name. Such variation makes it unclear whether models trained for the same bias category learn the same construct or different phenomena, a problem largely overlooked in prior work. We examine how definition choice affects bias annotation in a between-subjects experiment with 354 participants and a parallel evaluation with four LLMs. Participants and models rate six news articles across four bias categories using definitions that vary in conceptual framing and elaboration. Across 8,496 human and 28,800 LLM ratings, we find that the conceptual target of a definition drives annotation divergence, while construct-preserving elaboration does not: conceptual framing significantly shifts annotations for humans and does so even more strongly for LLMs. We discuss implications for construct specification in annotation protocols and prompt-based measurement, and consider how definitional sensitivity may propagate to downstream classification beyond media bias. We also release MUDD, the Multi-Definition Bias Detection Dataset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。