arXiv:2604.07419cs.IR2026-04被引 3

用视觉推理生成细粒度描述,提升文档检索精准度。

ReAlign: Optimizing the Visual Document Retriever with Reasoning-Guided Fine-Grained Alignment

  • 利用大模型推理生成查询相关的局部区域描述作为监督信号。
  • 在多个数据集上实现最高2%的相对性能提升,跨模型泛化性强。
  • 适合需要精准定位文档关键内容的场景,如法律、医疗文档检索。

视觉文档检索旨在从视觉丰富的文档集合中,根据查询找到相关页面。现有方法通常使用视觉语言模型(VLM)将查询和视觉页面编码到共享嵌入空间,并通过对比学习进行优化。然而,在视觉文档表征过程中,关键证据往往分散在复杂的文档布局中,导致检索模型难以捕捉有效线索。本文提出推理引导对齐(ReAlign),通过VLM的推理能力生成细粒度的视觉文档描述作为训练监督信号。具体而言,ReAlign使用高性能VLM识别页面中与查询相关的区域,并生成基于裁剪视觉区域的查询感知描述。随后,使用这些区域聚焦的描述来训练检索器,使由区域描述诱导的文档排序分布与原始查询诱导的分布对齐。在多样化的视觉丰富文档检索基准上的实验表明,ReAlign在域内和域外数据集上均持续提升性能,最高达2%相对改进。此外,该方法在不同VLM主干网络上均表现出良好泛化性,引导模型更关注关键视觉线索。代码与数据集已公开于https://github.com/NEUIR/ReAlign。

原文摘要 · Abstract (English)

Visual document retrieval aims to retrieve a set of document pages relevant to a query from visually rich collections. Existing methods often employ Vision-Language Models (VLMs) to encode queries and visual pages into a shared embedding space, which is then optimized via contrastive training. However, during visual document representation, localized evidence is usually scattered across complex document layouts, making it difficult for retrieval models to capture crucial cues for effective embedding learning. In this paper, we propose Reasoning-Guided Alignment (ReAlign), a method that enhances visual document retrieval by leveraging the reasoning capability of VLMs to provide fine-grained visual document descriptions as supervision signals for training. Specifically, ReAlign employs a superior VLM to identify query-related regions on a page and then generates a query-aware description grounding the cropped visual regions. The retriever is then trained using these region-focused descriptions to align the semantics between queries and visual documents by encouraging the document ranking distribution induced by the region-focused descriptions to match that induced by the original query. Experiments on diverse visually rich document retrieval benchmarks demonstrate that ReAlign consistently improves visual document retrieval performance on both in-domain and out-of-domain datasets, achieving up to 2% relative improvements. Moreover, the advantages of ReAlign generalize across different VLM backbones by guiding models to better focus their attention on critical visual cues for document representation. All code and datasets are available at https://github.com/NEUIR/ReAlign.

文档检索视觉语言模型细粒度对齐

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。