构建大规模结构化数据事实核查基准，挑战大模型真实推理能力
ClaimDB: A Fact Verification Benchmark over Large Structured Data
- 从百万级记录和多表数据中生成需程序推理的核查命题
- 30个大模型平均准确率低于55%，超半数表现不佳
- 揭示闭源与开源模型均难以识别无证据情况，适合高风险决策场景研究
现实中的事实核查常涉及大规模结构化数据。尽管已有诸多事实核查基准，但该场景仍被严重忽视。本文提出ClaimDB，一个基于数百万条记录和多张表格的结构化数据事实核查基准。其包含80个涵盖治理、医疗、媒体、教育及自然科学等领域的真实数据库。在此规模下，依赖文本阅读的验证方法失效，必须转向可执行程序推理。我们对30个主流闭源与开源（参数量小于700亿）大模型进行了广泛实验，发现超过一半模型准确率低于55%。分析还表明，无论闭源还是开源模型，在“放弃判断”（即承认无足够证据）能力上均表现薄弱，对其在高风险数据分析任务中的可靠性提出质疑。相关基准、代码与大模型排行榜已公开：https://claimdb.github.io。
原文摘要 · Abstract (English)
Real-world fact-checking often involves verifying claims grounded in structured data at scale. Despite substantial progress in fact-verification benchmarks, this setting remains largely underexplored. In this work, we introduce ClaimDB, a fact-verification benchmark where the evidence for claims is derived from compositions of millions of records and multiple tables. ClaimDB consists of 80 unique real-life databases covering a wide range of domains, from governance and healthcare to media, education and the natural sciences. At this scale, verification approaches that rely on "reading" the evidence break down, forcing a timely shift toward reasoning in executable programs. We conduct extensive experiments with 30 state-of-the-art proprietary and open-source (below 70B) LLMs and find that more than half score below 55% accuracy. Our analysis also reveals that both closed- and open-source models struggle with abstention -- the ability to admit that there is no evidence to decide -- raising doubts about their reliability in high-stakes data analysis tasks. We release the benchmark, code, and the LLM leaderboard at https://claimdb.github.io .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。