arXiv:2606.16092cs.CVcs.AI2026-06中稿 · CVPR

让大模型生成带图文引用的长回答,更真实还原文档信息。

VinQA: Visual Elements Interleaved Long-form Answer Generation for Real-World Multimodal Document QA

论文配图:VinQA: Visual Elements Interleaved Long-form Answer Generation for Real-World Multimodal Document QA
图 1 · 摘自论文原文
  • 用两种图像编码方式处理文档页,把图表等视觉元素作为可引用单位。
  • 引入新评估框架,发现微调后的开源模型表现接近闭源前沿模型。
  • 适合需要精准图文结合回答的研究者和开发者使用。

现实世界文档融合文本、表格、图表、照片和示意图,布局多样,但现有多模态大模型在文档问答中大多只输出纯文本,未能充分利用视觉元素。为此,我们提出VinQA数据集,支持长篇回答生成,并将引用的视觉元素明确插入对应文本中,且与原文档页面对齐。为支持该任务,我们研究了两种将原始文档页图像输入多模态大模型的方法及其视觉元素引用机制:(1)页面编码(Page Encoding),直接编码整页图像并标注视觉元素边界框,将其作为可引用单元;(2)模态编码(Modality Encoding),解析每页提取文本并裁剪视觉元素,分别编码,以裁剪出的元素为可引用单元。实验中,我们提出M-GroSE,一个扩展自GroUSE的多模态评估框架,从完整性、相关性、忠实度和不可回答性四个维度评估答案。同时报告视觉来源F1,直接衡量视觉引用准确率。尽管闭源前沿模型在测试集上仍领先,但对开源Qwen2.5-VL模型在训练集上微调后显著提升性能,缩小差距。初始阶段,模态编码在复杂文档(长文本、多视觉元素、多样化引用需求)中更鲁棒;微调后,页面编码达到相近水平,无需显式解析即可有效竞争。最后,基于多模态大模型的判别器Visual G-Eval证实,微调模型能在语义恰当位置插入视觉元素,并配以忠实支持文本。

原文摘要 · Abstract (English)

Real-world documents combine text with tables, charts, photographs, and diagrams arranged in diverse layouts, yet existing research on multimodal large language models (MLLMs) for document QA predominantly produces text-only responses, underutilizing these visual elements. We introduce VinQA, a dataset for long-form answer generation where cited visual elements are explicitly interleaved with their supporting text and grounded in relevant document pages. To support this task, we study two encoding methods for feeding raw document page images into an MLLM, along with their visual-element citation mechanisms: (1) Page Encoding, which directly encodes full-page images with bounding boxes of visual elements and treats these boxed regions as citable units; and (2) Modality Encoding, which parses each page to extract text and crop visual elements, encodes them separately, and uses these cropped elements as citable units. In our experiments, we propose M-GroSE, a multimodal evaluation framework extending GroUSE to assess answers along four dimensions: completeness, answer relevancy, faithfulness, and unanswerability. We additionally report Visual Source F1 to directly measure visual citation accuracy. Although proprietary frontier models still achieve the best overall scores on the VinQA test split, fine-tuning open Qwen2.5-VL models on the training split substantially improves their performance and narrows this gap. Modality Encoding is initially more robust for complex documents with long text, many visual elements, and diverse citation requirements. After training on VinQA, however, Page Encoding reaches a comparable level, competing effectively even without the explicit parsing used in Modality Encoding. Finally, Visual G-Eval, an MLLM-based judge, confirms that fine-tuned models insert visual elements at semantically appropriate positions with faithful supporting text.

文档问答图文生成多模态评估框架

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。