构建真实世界零样本事实核查检索基准，解决复杂声明的证据召回难题。
FactIR: A Real-World Zero-shot Open-Domain Retrieval Benchmark for Fact-Checking
- 基于生产日志与人工标注构建真实场景检索数据集
- 零样本下评估主流模型，揭示现有方法在复杂推理上的不足
- 适合从事事实核查、信息检索研究者参考
自动化事实核查日益依赖从网络获取证据以判断陈述的真实性。其核心挑战不仅在于检索相关信息，更在于发现能同时支持与反驳复杂陈述的证据。传统检索方法常返回直接回应陈述或偏向支持的文档，难以应对需间接推理的复杂声明。尽管已有部分基准和方法聚焦事实核查检索，但缺乏全面的真实世界开放域基准。本文提出基于Factiverse生产日志并经人工标注增强的现实检索基准FactIR，对先进检索模型在零样本设置下进行严格评估，并为构建实用的事实核查检索系统提供洞见。代码与数据已公开于https://github.com/factiverse/factIR。
原文摘要 · Abstract (English)
The field of automated fact-checking increasingly depends on retrieving web-based evidence to determine the veracity of claims in real-world scenarios. A significant challenge in this process is not only retrieving relevant information, but also identifying evidence that can both support and refute complex claims. Traditional retrieval methods may return documents that directly address claims or lean toward supporting them, but often struggle with more complex claims requiring indirect reasoning. While some existing benchmarks and methods target retrieval for fact-checking, a comprehensive real-world open-domain benchmark has been lacking. In this paper, we present a real-world retrieval benchmark FactIR, derived from Factiverse production logs, enhanced with human annotations. We rigorously evaluate state-of-the-art retrieval models in a zero-shot setup on FactIR and offer insights for developing practical retrieval systems for fact-checking. Code and data are available at https://github.com/factiverse/factIR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。