跨数据集评估九种自动验真模型,发现检索仍是最大瓶颈。
How Robust Are Automated Fact-Checking Systems? A Cross-Benchmark Evaluation

- 在多领域数据上测试九类模型,涵盖从随机基线到微调Transformer的全链条方法
- 替换为真实证据后准确率提升14-22点,证明检索环节严重拖累整体性能
- 不同领域和评价指标下模型排名差异巨大,现有系统泛化能力极弱
自动化事实核查(AFC)系统通过检索证据并预测陈述真实性来工作,但现有评估忽略了简单基线,且多数系统仅针对单一基准训练,难以跨领域推广。此前研究未在多样数据集上对完整的‘检索-验证’两阶段流程进行交叉评估,既补充了仅关注检索的研究(Thakur et al., 2021),也超越了单阶段基准研究(Calamai et al., 2025)。本文在四个覆盖科学、开放网络和气候领域的数据集上,对九种模型(包括随机、稀疏基线、微调Transformer、零样本大模型,以及AVeriTeC 2025竞赛中排名最高的两个系统)进行了全面评测。主要发现:(1)在ClimateCheck仅用陈述的数据上,微调模型优于零样本大模型及顶尖的AVeriTeC 2025系统,表明噪声证据会削弱判断准确性;(2)系统排名高度依赖于领域和评估指标——在SciFact上表现最佳的模型(宏平均F1为0.70)在ClimateCheck上下降至0.31,而AVeriTeC 2025冠亚军在不同指标与数据集下互换排名;(3)将检索结果替换为黄金标注证据后,各模型的真伪判断准确率普遍提升14–22个百分点,证实检索环节仍是核心瓶颈。代码、预处理数据集和全部结果均已公开,以支持可复现的事实核查研究。
原文摘要 · Abstract (English)
Automated fact-checking (AFC) systems retrieve evidence and predict claim veracity, yet evaluations omit simple baselines, systems are developed for a single benchmark and cannot be trusted to generalise across domains. No prior work cross-evaluates the full two-stage retrieve-then-verify pipeline across diverse datasets, complementing retrieval-only studies (Thakur et al., 2021) and single-stage benchmarking studies (Calamai et al., 2025). We benchmark nine models, ranging from random and sparse baselines to fine-tuned transformers, zero-shot LLMs, and the two highest-ranked systems from the AVeriTeC 2025 shared task, across four datasets spanning scientific, open-web, and climate domains. Three findings stand out: (1) on ClimateCheck claim-only and fine-tuned models outperform zero-shot LLM and top-performing AVeriTeC 2025 systems, highlighting that noisy evidence can degrade veracity prediction; (2) system rankings are strongly domain- and metric-dependent: the best model on SciFact (macro-F1 0.70) drops to 0.31 on ClimateCheck, while the AVeriTeC 2025 winner and runner-up swap rankings based on evaluation metrics and datasets; (3) replacing retrieved evidence with gold annotations improves veracity accuracy by 14-22 points across models, confirming retrieval remains primary bottleneck. We release code, pre-processed datasets, and all results to support reproducible AFC research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。