为文档智能设计可追踪证据的评测基准，揭示模型答对题却引错段落的问题。
CiteVQA: Benchmarking Evidence Attribution for Trustworthy Document Intelligence
- 要求模型返回答案和对应的精准段落位置，联合评估正确性
- 20个大模型测试中最强仅76%答案与引用同时正确，开源模型不足23%
- 适用于法律、医疗等需严格溯源的高风险领域研究者
多模态大语言模型虽显著提升文档理解能力，但现有文档问答评测仅关注最终答案，忽略支撑证据的可靠性。这种只看答案的评估方式掩盖了关键缺陷：模型可能得出正确答案，却基于错误段落推理——在法律、金融、医疗等高风险领域极危险。为此，我们提出CiteVQA基准，要求模型在回答问题时同时提供元素级边界框引用，实现答案与证据的联合评估。该基准包含711份PDF文档，覆盖7个领域和两种语言，共1,897个问题，平均每份文档40.6页。真实引用通过掩码消融自动识别，并经专家审核验证。核心评估指标为严格归因准确率（SAA），仅当答案和引用区域均正确才计分。对20个MLLM的审计显示普遍存在归因幻觉：多数模型答对但引错。最强系统Gemini-3.1-Pro-Preview的SAA为76.0%，最强开源模型仅22.5%。CiteVQA揭示了传统评估忽略的可靠性缺口，为构建可信文档智能提供了必要工具。代码库已公开于https://github.com/opendatalab/CiteVQA。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have significantly advanced document understanding, yet current Doc-VQA evaluations score only the final answer and leave the supporting evidence unchecked. This answer-only approach masks a critical failure mode: a model can land on the correct answer while grounding it in the wrong passage -- a critical risk in high-stakes domains like law, finance, and medicine, where every conclusion must be traceable to a specific source region. To address this, we introduce CiteVQA, a benchmark that requires models to return element-level bounding-box citations alongside each answer, evaluating both jointly. CiteVQA comprises 1,897 questions across 711 PDFs spanning seven domains and two languages, averaging 40.6 pages per document. To ensure fidelity and scalability, the ground-truth citations are generated by an automated pipeline-which identifies crucial evidence via masking ablation-and are subsequently validated through expert review. At the core of our evaluation is Strict Attributed Accuracy (SAA), which credits a prediction only when the answer and the cited region are both correct. Auditing 20 MLLMs reveals a pervasive Attribution Hallucination: models frequently produce the right answer while citing the wrong region. The strongest system (Gemini-3.1-Pro-Preview) achieves an SAA of only 76.0, and the strongest open-source MLLM reaches just 22.5. Ultimately, towards trustworthy document intelligence, CiteVQA exposes a reliability gap that answer-only evaluations overlook, providing the instrumentation needed to close it. Our repository is available at https://github.com/opendatalab/CiteVQA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。