无需推理的多模态稀疏检索,让视觉文档搜索更快更准。
Inference-Free Multimodal Learned Sparse Retrieval for Production-Scale Visual Document Search

- 用图像标题作为关键词提示,训练时激活相关词汇维度。
- 在1870万文档上,召回率比同类模型提升一倍以上。
- 适合生产级部署,既快又准,无需实时神经编码。
随着arXiv论文和企业PDF等大规模视觉文档语料持续增长,视觉文档检索日益重要;然而,仍缺乏可在生产环境中部署、无需神经编码即可服务查询的可扩展系统。现有方法要么依赖基于视觉语言模型(VLM)的密集或多向量模型实现优异检索效果,但需在服务时进行神经编码;要么采用基于OCR或生成标题的BM25方法避免查询编码,却面临耗时的文本提取或生成。为填补这一空白,我们提出V-SPLADE,一种无需推理的视觉文档稀疏检索器。然而,此类免推理的多模态学习稀疏检索系统仍处于探索阶段,尚未在高稀疏度下达到密集模型的性能水平。我们将其归因于词汇定位问题:视觉稀疏表示往往无法捕捉图像中的词汇内容。为此,我们引入“标题门控词元监督”——一种仅在训练时使用的信号,利用VLM生成的标题作为词汇线索,激活与检索相关的词元维度。通过该监督机制,V-SPLADE在六个视觉文档检索基准上,平均NDCG@5相比同规模密集基线提升+13.8个百分点,相比基于OCR或标题的BM25基线最高提升+6.3个百分点。在1870万文档语料上,其R@5超过同规模密集基线两倍以上,并通过分数融合使竞争性检索器的R@5进一步提升+2.4个百分点。代码将很快发布于https://github.com/naver/v-splade。
原文摘要 · Abstract (English)
As large-scale visual-document corpora such as arXiv papers and enterprise PDFs continue to grow, visual-document retrieval has gained increasing attention; yet it still lacks a deployable system that lexically indexes visual documents to serve queries without neural encoding at scale. Existing methods either achieve strong retrieval quality with VLM-based dense or multi-vector models but require neural query encoding at serving time, or avoid query encoding with OCR- or caption-based BM25 at the cost of time-consuming text extraction or generation. To fill this missing serving regime, we present V-SPLADE, an inference-free sparse retriever for visual-document retrieval. However, such inference-free multimodal learned sparse retrieval systems remain underexplored and have not yet shown dense-level effectiveness under high sparsity. We attribute this limitation to a lexical grounding problem: visual sparse representations often fail to capture the lexical content embedded in document images. To address this problem, we introduce caption-gated token supervision, a training-only signal that uses VLM-generated captions as lexical cues to activate retrieval-relevant vocabulary dimensions. With this supervision, V-SPLADE improves average NDCG@5 across six visual-document retrieval benchmarks by +13.8pp over the same-scale dense baseline and by up to +6.3pp over OCR- or caption-based BM25 baselines. On an 18.7M-document corpus, it more than doubles R@5 over the same-scale dense baseline and further improves competing retrievers through score fusion by up to +2.4pp R@5. Code will be released soon at https://github.com/naver/v-splade.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。