构建媒体背景核查数据集，让AI学会判断信息来源可信度。
Generating Media Background Checks for Automated Source Critical Reasoning
- 从媒体偏见网站提取6709条背景核查数据，构建新NLP任务
- 检索增强模型在该任务上表现显著提升，准确率明显改善
- 结果对人类和模型都有帮助，适合做信息可信度评估
网络上的信息并不总是真实。这一现实要求人类和模型在处理检索到的信息时进行复杂的可信度推理。然而，自然语言处理领域对此关注甚少——检索增强模型通常不被期待质疑检索文档的可靠性。人类专家通过收集关于来源的背景、可靠性和倾向性的信号来克服这一挑战，即进行源批判。本文提出一项新的NLP任务，聚焦于发现并总结这些信号。我们引入了一个新数据集，包含6,709条来自“Media Bias / Fact Check”网站的“媒体背景检查”记录，该网站由志愿者运营，用于记录媒体偏见情况。我们在该数据集上测试了开源与闭源大模型基线，并对比了有无检索的情况，发现检索显著提升了性能。此外，我们进行了人工评估，证明：1）媒体背景检查对人类有帮助；2）媒体背景检查对检索增强模型也有帮助。
原文摘要 · Abstract (English)
Not everything on the internet is true. This unfortunate fact requires both humans and models to perform complex reasoning about credibility when working with retrieved information. In NLP, this problem has seen little attention. Indeed, retrieval-augmented models are not typically expected to distrust retrieved documents. Human experts overcome the challenge by gathering signals about the context, reliability, and tendency of source documents - that is, they perform source criticism. We propose a novel NLP task focused on finding and summarising such signals. We introduce a new dataset of 6,709 "media background checks" derived from Media Bias / Fact Check, a volunteer-run website documenting media bias. We test open-source and closed-source LLM baselines with and without retrieval on this dataset, finding that retrieval greatly improves performance. We furthermore carry out human evaluation, demonstrating that 1) media background checks are helpful for humans, and 2) media background checks are helpful for retrieval-augmented models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。