延迟视觉处理，先索引后分析，提升工程文档问答准确率
Index Light, Reason Deep: Deferred Visual Ingestion for Visual-Dense Document Question Answering
- 预处理时不调用视觉模型，仅用目录和图号构建分层索引
- 在桥涵图纸上问答准确率达65.6%，比传统方法高41.3个百分点
- 适合处理图像密集、相似度高的工程文档，尤其适合工业场景
现有多模态文档问答方法普遍采用预摄入（PI）策略：在索引阶段，对每页调用视觉语言模型（VLM）生成描述并编码为向量，通过嵌入相似性检索回答问题。然而该方法在图像密集的工程文档中面临双重困境：VLM生成的盲描述会丢失关键视觉细节，且嵌入检索在高度相似文档上系统性失效。本文提出延迟视觉摄入（DVI）框架：预处理阶段零次调用VLM，仅利用文档结构信息（目录、图号）通过HDNC（分层图号聚类）算法自动构建分层索引；推理时先用BM25检索候选页面，再将原图与具体问题送入VLM进行精准分析。在三个数据集上的大规模实验验证了DVI的有效性：在桥涵工程图纸（1,323个问题）上，端到端问答准确率达65.6%（对比PI的24.3%，+41.3pp）；在钢构目录（186个问题）上达30.6%（对比16.1%，+14.5pp）；在CircuitVQA公开基准（9,315个问题）上，检索ImgR@3达31.2%（对比0.7%）。对同为视觉检索领先方法的ColPali（ICLR 2025 SOTA）在桥涵数据集上仅得20.1% PageR@3，表明其失败是结构性的，非模型能力不足。消融实验证明，零成本的HDNC索引带来+27.5pp检索提升，且VLM调用率分析确认瓶颈在检索侧而非理解侧。
原文摘要 · Abstract (English)
Existing multimodal document question answering methods predominantly adopt a Pre-Ingestion (PI) strategy: during the indexing phase, a Vision Language Model (VLM) is called on every page to generate page descriptions that are then encoded into vectors, and questions are answered via embedding similarity retrieval. However, this approach faces a dual dilemma on visual-dense engineering documents: VLM blind descriptions inevitably lose critical visual details, and embedding retrieval systematically fails on highly similar documents. This paper proposes the Deferred Visual Ingestion (DVI) framework: zero VLM calls during preprocessing, leveraging only document structural information (table of contents, drawing numbers) to automatically build a hierarchical index through the HDNC (Hierarchical Drawing Number Clustering) algorithm; during inference, candidate pages are located via BM25 retrieval, and the original images along with the specific question are sent to a VLM for targeted analysis. Large-scale experiments on three datasets validate the effectiveness of DVI: on Bridge engineering drawings (1,323 questions), end-to-end QA accuracy reaches 65.6\% vs. PI's 24.3\% (+41.3pp); on Steel catalog (186 questions), 30.6\% vs. 16.1\% (+14.5pp); on CircuitVQA, a public benchmark (9,315 questions), retrieval ImgR@3 achieves 31.2\% vs. 0.7\%. On the Bridge dataset, we evaluated ColPali (ICLR 2025 visual retrieval SOTA), which achieved only 20.1\% PageR@3, demonstrating that the failure of embedding retrieval on homogeneous engineering documents is structural rather than due to insufficient model capability. Ablation studies show that HDNC zero-cost automatic indexing yields a +27.5pp retrieval improvement, and VLM conversion rate analysis confirms that the bottleneck lies on the retrieval side rather than the comprehension side.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。