构建公开媒体可信度核查数据集，支持低成本可复现的自动事实核查研究。
MEDIAREF: A Public Knowledge Store for Media Background Checks
- 构建200家媒体的公开知识库，支持可复现的媒体背景核查生成。
- 评估主流大模型在媒体可信度分析任务上表现，发现其生成质量受限于数据来源。
- 替代昂贵私有搜索接口，为研究者提供低成本、透明的评测资源。
基于大语言模型的检索增强生成（RAG）在自动化事实核查（AFC）中应用日益广泛。通过将模型输出与检索到的证据结合，RAG系统能提供透明的推理依据，并独立于模型更新外部信息。然而，现有方法通常假设检索内容可靠，而真实信息可能存在冲突、过时或来自不可靠、偏见源。近期工作提出“源批判推理”（source-critical reasoning），通过媒体背景检查（MBC）评估证据来源可信度以支持事实验证。但生成MBC需依赖昂贵的专有搜索接口，制约可复现性。为此，我们提出MEDIAREF——一个面向200家媒体的公开网页文档知识库，支持低成本、可复现的MBC生成评估。本文描述了可复现的构建与更新方法，评估了多种主流大模型在该任务上的表现，并通过自动与定性评估证明，MEDIAREF能显著提升MBC生成质量。
原文摘要 · Abstract (English)
LLM-based retrieval-augmented generation (RAG) is increasingly used for automated fact-checking (AFC) and related tasks. By grounding LLM outputs in retrieved evidence, RAG-based systems provide transparent justifications while allowing external information to be updated independently of the underlying model. However, existing approaches often assume retrieved evidence is reliable, although real-world information may be conflicting, outdated, and can originate from unreliable or biased sources. Recent work on *source-critical reasoning* addresses this challenge through media background checks (MBCs) (Schlichtkrull, 2024), which assess the credibility of evidence sources to support downstream fact verification. However, generating MBCs relies on costly proprietary search APIs, limiting reproducibility. To mitigate this issue, we introduce MEDIAREF, a publicly available knowledge store of web-sourced documents that enables reproducible, low-cost evaluation of MBC generation across 200 media sources. We describe a reproducible methodology for constructing and updating the collection, assess widely used LLMs on the MBC generation task, and demonstrate that MEDIAREF supports higher-quality MBC generation through both automatic and qualitative evaluation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。