将视觉相似度与文本坐标结合,精准定位文档中的关键信息区域。
Spatially-Grounded Document Retrieval via Patch-to-Region Relevance Propagation
- 用视觉块相似度筛选OCR提取的文本区域,实现空间相关性过滤。
- 在指定页面内定位准确率高达59.7%([email protected]),比随机选择提升近9倍。
- 无需训练即可运行,适合需要精确定位的生成型检索场景。
ColPali等晚期交互多模态检索模型通过将文档页作为图像嵌入,并计算查询词元与视觉块之间的细粒度相似度,实现了最先进的文档检索效果。然而,这些方法仅在页面级别操作,限制了在检索增强生成(RAG)中对精确上下文的需求。相反,基于OCR的系统虽能提取带坐标位置的结构化文本,但缺乏语义上的相关性判断依据。本文提出一种混合架构,将ColPali的块级相似度得分作为对OCR提取区域的空间相关性过滤器。我们建立了视觉变换器块网格与OCR边界框之间的坐标映射关系,引入交集度量进行相关性传播,并给出面积效率的理论边界。在包含真实边界框的BBox-DocVQA数据集上评估:当页面检索正确时,ColQwen3-4B配合百分位50阈值的命中率为59.7%([email protected]),84.4%([email protected]),35.8%([email protected]),平均IoU达0.569,远超随机区域选择的约6.7%。相比返回全部OCR区域,该方法减少28.8%上下文令牌;相比全页图像令牌,减少52.3%。本方法在推理阶段运行,无需额外训练。代码已开源:https://github.com/athrael-soju/Snappy。
原文摘要 · Abstract (English)
Late-interaction multimodal retrieval models like ColPali achieve state-of-the-art document retrieval by embedding pages as images and computing fine-grained similarity between query tokens and visual patches. However, they operate at page-level granularity, limiting utility for retrieval-augmented generation (RAG) where precise context is paramount. Conversely, OCR-based systems extract structured text with bounding box coordinates but lack semantic grounding for relevance assessment. We propose a hybrid architecture that unifies these paradigms: using ColPali's patch-level similarity scores as spatial relevance filters over OCR-extracted regions. We formalize the coordinate mapping between vision transformer patch grids and OCR bounding boxes, introduce intersection metrics for relevance propagation, and establish theoretical bounds on area efficiency. We evaluate on BBox-DocVQA with ground-truth bounding boxes. For within-page localization (given correct page retrieval), ColQwen3-4B with percentile-50 thresholding achieves 59.7% hit rate at [email protected] (84.4% at [email protected], 35.8% at [email protected]), with mean IoU of 0.569, compared to ~6.7% for random region selection. Our approach reduces context tokens by 28.8% compared to returning all OCR regions and by 52.3% compared to full-page image tokens. Our approach operates at inference time without additional training. We release Snappy, an open-source implementation at https://github.com/athrael-soju/Snappy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。