让文档布局可学习，提升跨异构版式文档的检索准确率
Beyond Bag-of-Patches: Learning Global Layout via Textual Supervision for Late-Interaction Visual Document Retrieval
- 用文本描述监督训练全局布局嵌入，增强局部图像块表示
- 在四个数据集上比最强基线提升2.4 nDCG@5和2.3 MAP@5
- 适合需要理解复杂排版结构的文档检索场景
视觉文档检索（VDR）模型多采用后期交互架构，将文档表示为局部图像块嵌入并匹配查询令牌。此类架构虽高效，但更关注局部相似性，忽视文档的全局版式结构，导致在包含图表、表格与文字混合布局的文档中产生误差。本文提出一种多模态编码器，在不改变推理流程的前提下，通过文本描述编码的文档布局信息，为局部图像块嵌入添加全局布局嵌入。在四个ViDoRe-v2数据集上，该模型相较最强基线ColPali/ColQwen在nDCG@5上提升+2.4，在MAP@5上提升+2.3，且各数据集均具统计显著性。
原文摘要 · Abstract (English)
Visual Document Retrieval (VDR) models mostly rely on late interaction architectures, in which documents are represented by a set of local patch embeddings and then matched against query tokens. While efficient, this architecture prioritizes local similarity over global layout structure of documents to estimate relevancy between documents and query. In practice, this leads to errors as relevance originates from layout structure of documents with heterogeneous layouts combining figures, tables, and text. We make document layout learnable without changing inference. We propose a multimodal encoder that augments local patch representations with a global layout embedding, trained via textual descriptions encoding document layout information. Across four ViDoRe-v2 datasets, our model improves over the strongest architecturally comparable ColPali/ColQwen baseline by +2.4 nDCG@5 and +2.3 MAP@5, with statistically significant per-dataset gains over ColQwen.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。