让多模态文档检索更快更省内存,适合大模型落地应用。
Stellar: Scalable Multimodal Document Retrieval for Natural Language Queries

- 用稀疏编码器筛选候选文档,减少需加载的嵌入数量。
- 在磁盘存储嵌入向量,仅动态加载必要部分,内存降低1-2个数量级。
- 适用于需要高效检索的大规模真实场景,如RAG系统部署。
多模态文档检索在检索增强生成(RAG)系统中至关重要,需从大规模语料库中选出最相关的多模态文档回答自然语言查询。现有方法使用多向量表示文档与查询,并通过后期交互实现高精度,但带来巨大内存开销,影响可扩展性与实际部署。本文提出Stellar框架,将令牌级文档嵌入存储于磁盘,仅将少量候选嵌入加载至内存进行后期交互。其核心包含两部分:(i) 基于词汇表示的过滤(LRF),微调多模态大模型作为稀疏编码器生成高质量词汇表示,实现高效文档筛选,大幅缩小候选集;(ii) 高效磁盘后端后期交互(DLI),设计基于平衡聚类算法的磁盘嵌入存储布局,结合简单有效的成本模型,动态加载所需令牌嵌入。在四个真实世界基准和一个新提出的超大规模数据集上的实验表明,Stellar在不损失检索效果的前提下,将内存开销和查询延迟降低1-2个数量级。
原文摘要 · Abstract (English)
Multimodal document retrieval--selecting the most relevant multimodal document from a large corpus to answer a natural language query--plays an essential role in Retrieval-Augmented Generation (RAG) systems. State-of-the-art methods represent each document and query with multiple token-level embeddings and use late interaction to achieve high effectiveness. However, such multi-vector representations incur substantial memory overhead during retrieval, leading to poor scalability and hindering real-world deployment. In this paper, we present Stellar, a scalable multimodal document retrieval framework that stores token-level document embeddings on disk and loads only a small set of candidate embeddings into memory for late interaction. Stellar comprises two key components: (i) Lexical Representation-based Filtering (LRF), which fine-tunes a Multimodal Large Language Model (MLLM) as a sparse encoder to produce high-quality lexical representations, enabling efficient and effective document filtering to significantly reduce the candidate set; (ii) Efficient Disk-backed Late Interaction (DLI), which designs an on-disk token embedding storage layout guided by a balanced clustering algorithm, and dynamically loads only the necessary token embeddings into memory using a simple yet effective cost model. Extensive experiments on four real-world benchmarks and a newly presented large-scale dataset demonstrate that Stellar reduces memory overhead and query latency by 1-2 orders of magnitude compared to existing methods without compromising retrieval effectiveness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。