MetaEmbed让多模态检索在测试时可灵活扩展,兼顾精度与效率。
MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interaction
- 用可学习的元标记生成多向量嵌入,支持测试时动态调整数量
- 在MMEB和ViDoRe上达到顶尖检索性能,支持320亿参数模型
- 适合需要灵活平衡速度与精度的多模态检索系统
通用多模态嵌入模型在捕捉查询与候选项间语义相关性方面取得了巨大成功。然而,现有方法要么将查询与候选项压缩为单一向量,可能限制细粒度信息表达;要么生成过多向量,导致多向量检索开销过大。本文提出MetaEmbed框架,重新思考大规模多模态嵌入的构建与交互方式。训练时,在输入序列中添加固定数量的可学习元标记(Meta Tokens);测试时,其最后一层上下文表示作为紧凑而丰富的多向量嵌入。通过提出的马特罗什卡多向量检索训练策略,MetaEmbed学会按粒度组织信息于多个向量中。结果实现了测试时的可扩展检索:用户可根据需求选择用于索引和检索交互的标记数量,平衡检索质量与效率。在大规模多模态嵌入基准(MMEB)和视觉文档检索基准(ViDoRe)上的大量评估表明,MetaEmbed在320亿参数模型下仍保持卓越检索表现,并具备强可扩展性。代码已开源。
原文摘要 · Abstract (English)
Universal multimodal embedding models have achieved great success in capturing semantic relevance between queries and candidates. However, current methods either condense queries and candidates into a single vector, potentially limiting the expressiveness for fine-grained information, or produce too many vectors that are prohibitive for multi-vector retrieval. In this work, we introduce MetaEmbed, a new framework for multimodal retrieval that rethinks how multimodal embeddings are constructed and interacted with at scale. During training, a fixed number of learnable Meta Tokens are appended to the input sequence. At test-time, their last-layer contextualized representations serve as compact yet expressive multi-vector embeddings. Through the proposed Matryoshka Multi-Vector Retrieval training, MetaEmbed learns to organize information by granularity across multiple vectors. As a result, we enable test-time scaling in multimodal retrieval where users can balance retrieval quality against efficiency demands by selecting the number of tokens used for indexing and retrieval interactions. Extensive evaluations on the Massive Multimodal Embedding Benchmark (MMEB) and the Visual Document Retrieval Benchmark (ViDoRe) confirm that MetaEmbed achieves state-of-the-art retrieval performance while scaling robustly to models with 32B parameters. Code is available at https://github.com/facebookresearch/MetaEmbed.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。