让AI能自动检查并修正文档页码引用错误,提升问答可信度。
AtomCite: Verification and Correction of Supplied Page-Level Citations in Multi-Page Documents

- 将答案拆解为观点,比对引用页图像进行验证与修复。
- 在注入错误数据上达到93%验证准确率,修复后引用精确率达87%-90%。
- 无需训练即可迁移使用,适合需要高可信引用的文档问答系统。
大型语言模型在多页文档问答时应引用支持页,但现有引用常不准确。当前评估仅在生成时或文本片段层面进行,缺乏对已附页码引用的验证与修正能力评估。本文提出AtomCite,一种解析答案为论点、比对引用页图像并执行确定性修复策略的智能体框架。为此构建了DocCite,据知是首个针对文档图像中页码引用验证与修正的基准,基于MP-DocVQA和DUDE,包含928个人工验证的注入错误与2,468个从前沿及高效模型中收集的候选自然错误,经双标注员审计确认1,909个为真实错误。主标签由人工确定,非依赖LLM判断,另设独立验证层。在Gemini、Claude和GPT三类模型上,AtomCite在注入基准上实现约93%的二分类验证准确率,显著优于所有仅用OCR的条件,包括计算匹配对照组,并超越所有先前基于文本的基线(相同OCR输入)。其修复策略使混合错误集的引用精确率从原始34%提升至87%-90%,同时保留超90%正确引用。该框架可零样本迁移:固定提示、零训练,使两个开源7-8B模型在五个公开基准上的幻觉检测得分超越直接提示为裁判的版本。最终审计表明,自动标签噪声会扭曲验证器准确率,甚至反转系统排名,依赖合成或自动标签的评估可能误判验证能力。
原文摘要 · Abstract (English)
Large language models answering questions over multi-page documents are expected to cite the supporting pages, yet supplied citations are sometimes inaccurate, and current evaluations score citations at generation time or against text passages: no existing benchmark evaluates whether a system can verify and correct a page-level citation already attached to an answer. We propose AtomCite, an agentic framework that parses an answer into claims, checks each claim against the image of its cited page, and applies a deterministic repair policy. To evaluate it, we introduce DocCite, to our knowledge the first benchmark for systems that verify and correct page-level citations in document images. Built on MP-DocVQA and DUDE, it combines 928 validated injected instances with 2,468 candidate natural errors harvested from frontier- and efficiency-tier models, of which a two-annotator audit confirms 1,909 as genuine errors. Primary labels are assigned deterministically, not by LLM judges, with the human audit as a separate validation layer. Across three model families (Gemini, Claude, and GPT), AtomCite reaches around 93% binary verification accuracy on the injected benchmark, significantly outperforming every OCR-only condition, including a compute-matched control, and exceeding every prior text-based baseline given the same OCR text. Its repair policy lifts citation precision on the injected mix from a constructed 34% to 87-90% while retaining over 90% of correct claims. AtomCite also transfers: with frozen prompts and zero training, it raises the hallucination-detection scores of two open 7-8B models on five public benchmarks above the same models prompted as direct judges. Finally, the audit shows that noise in automatic labels biases measured verifier accuracy and can reverse system rankings, so evaluations relying only on synthetic or automatic labels risk mismeasuring verification capability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。