提出上下文感知的多模态文档检索框架,提升跨页信息理解能力。
CMDR: Contextual Multimodal Document Retrieval

- 联合编码多页内容,从共享上下文表示中生成页面嵌入
- 在新基准上显著优于传统独立编码方法,提升跨页查询准确率
- 适合需要理解长文档结构与跨页关联的应用场景
多模态文档检索旨在保留原文档文本与视觉内容的同时召回相关页面。然而现有评测主要依赖简单的词法或语义匹配,多数方法独立编码页面,忽略了需跨页整合信息的查询所依赖的文档上下文。本文提出CMDR任务与CMDR-Bench基准,要求建模文档上下文信息。为此,我们设计了CMDR-Embed框架,通过联合编码多页内容,从共享上下文表示中提取页面级嵌入。同时引入CMCL对比学习目标,平衡上下文建模与页面判别性。实验表明,CMDR-Embed显著优于非上下文嵌入,验证了上下文感知多模态嵌入对文档检索的重要性。
原文摘要 · Abstract (English)
Multimodal document retrieval aims to retrieve relevant pages while preserving both textual and visual content from the original document. However, existing benchmarks primarily evaluate simple lexical or semantic matching, and most methods encode pages independently. Consequently, they overlook the contextual information in the document required to resolve queries that aggregate information across multiple pages. In this paper, we introduce CMDR and CMDR-Bench, a new multimodal document retrieval task and benchmark that require modeling document context. To address this challenge, we propose CMDR-Embed, a contextual multimodal embedding framework that explicitly incorporates document context by jointly encoding multiple pages and deriving page-level embeddings from a shared contextual representation. Furthermore, we introduce CMCL, a contextual multimodal contrastive learning objective that effectively trains CMDR-Embed by balancing contextual modeling with page-level discriminability. Experiments demonstrate that CMDR-Embed significantly outperforms non-contextual embeddings, highlighting the importance of context-aware multimodal embeddings for advancing document retrieval.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。