构建动态多语言虚假信息检测基准，揭示模型真实表现与人类共识的差距。
CommunityFact: A Dynamic, Multilingual, Multi-domain Benchmark for Misinformation Detection in the Wild
- 基于社区笔记构建可更新的跨语言、跨领域虚假信息验证数据集
- 网页检索使模型性能提升显著，但选源策略与人类共识不一致
- 适合研究多语言信息验证、模型可信度评估及训练信号设计的学者
虚假信息验证正发生在公开、快速变化且多语言的在线环境中，静态基准难以全面评估模型可靠性。我们提出CommunityFact，一个可刷新的野外虚假信息检测基准，旨在实现覆盖广度、粒度精细和可重分配性。当前版本包含15,992条独立声明，涵盖五种语言和两个领域。我们在十种大语言模型上评估了不同推理能力下的表现，包括思维链和网页搜索。结果表明：闭源输入验证仍具挑战性，网页访问带来最大提升；而开启网页的LLM在来源选择上系统性偏离人类社区笔记评审者达成的共识——这一差距可通过模型特定的检索扩展或剪枝机制缩小。此外，不同语言-领域组合间差异显著，且网络型系统所依赖的证据生态也存在明显异质性。除评估外，CommunityFact还将社区笔记定位为条件化来源推荐器的训练信号，有望提升对新声明的事实验证能力。
原文摘要 · Abstract (English)
Misinformation verification increasingly occurs in public, fast-moving, and multilingual online settings, where static benchmarks provide an incomplete measure of model reliability. We introduce CommunityFact, a refreshable benchmark for misinformation detection in the wild, with three major goals: coverage, granularity, and redistributability. This release contains 15,992 standalone claims across five languages and two domains. We evaluate ten LLMs under varying inference-time capabilities, including thinking and web-search. Our results show that closed-input verification remains challenging, web access yields the largest gains, and web-enabled LLMs' source-selection policies are systematically misaligned with the sources human Community Notes raters converge on -- a gap that closes through model-specific mechanisms of retrieval expansion or pruning. We further find substantial variation across language-domain slices and across the evidence ecosystems used by web-enabled systems. Beyond evaluation, CommunityFact positions Community Notes as a training signal for claim-conditioned source suggesters that could improve factual verification on novel claims.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。