arXiv:2605.08888cs.CLcs.CV2026-05被引 1

评测大模型在长文档中可信推理能力,揭示答案正确但证据链不全的严重问题。

DocScope: Benchmarking Verifiable Reasoning for Trustworthy Long-Document Understanding

论文配图:DocScope: Benchmarking Verifiable Reasoning for Trustworthy Long-Document Understanding
图 1 · 摘自论文原文
  • 将长文档问答转化为分阶段推理轨迹预测,逐级验证证据定位、区域标注、事实提取和答案验证。
  • 仅29%正确答案拥有完整证据链,区域标注是各环节中最薄弱的步骤。
  • 适合关注可解释性、可信AI评估的研究者与开发者,尤其关注多模态长文档理解。

评估多模态大模型在长篇图文文档上生成可信、可验证推理的能力,需超越端到端答案准确率。我们提出DocScope基准,将长文档问答建模为结构化推理轨迹预测任务:给定完整PDF文档和问题,模型输出证据页、支持性区域、相关事实陈述及最终答案。设计四阶段评估协议——页定位、区域标注、事实提取、答案验证——通过阶段解耦独立审计每一步,所有评审员均经人类对齐研究筛选与校准。DocScope包含1,124个问题,源自273份文档,所有层级证据标注均由人工完成。我们评测6个私有模型、12个开源权重模型及多个领域专用系统。实验表明,答案准确率无法替代轨迹级评估:即使答案正确,完整证据链的最高观测比例仅为29%。所有模型中,区域标注仍是最弱环节。主要难点在于整合分散于长距离和多个文档簇中的证据;而理想模型研究揭示,忠实感知与事实提取是核心能力瓶颈。跨架构对比进一步表明,激活参数量比总规模更具决定性影响。基准与代码将公开发布于https://github.com/MiliLab/DocScope。

原文摘要 · Abstract (English)

Evaluating whether Multimodal Large Language Models can produce trustworthy, verifiable reasoning over long, visually rich documents requires evaluation beyond end-to-end answer accuracy. We introduce DocScope, a benchmark that formulates long-document QA as a structured reasoning trajectory prediction problem: given a complete PDF document and a question, the model outputs evidence pages, supporting evidence regions, relevant factual statements, and a final answer. We design a four-stage evaluation protocol -- Page Localization, Region Grounding, Fact Extraction, and Answer Verification -- that audits each level of the trajectory independently through inter-stage decoupling, with all judges selected and calibrated via human alignment studies. DocScope comprises 1,124 questions derived from 273 documents, with all hierarchical evidence annotations completed by human annotators. We benchmark 6 proprietary models, 12 open-weight models, and several domain-specific systems. Our experiments reveal that answer accuracy cannot substitute for trajectory-level evaluation: even among correct answers, the highest observed rate of complete evidence chains is only 29\%. Across all models, region grounding remains the weakest trajectory stage. Furthermore, the primary difficulty stems from aggregating evidence dispersed across long distances and multiple document clusters, while an oracle study identifies faithful perception and fact extraction as the dominant capability bottleneck. Cross-architecture comparisons further suggest that activated parameter count matters more than total scale. The benchmark and code will be publicly released at https://github.com/MiliLab/DocScope.

可信推理长文档理解多模态评估基准

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。