让普通模型变高效多向量检索器,无需重训练。
Your Embedding Model is SMARTer Than You Think

- 利用隐藏层特征做后期交互,不改原模型结构。
- 在多个数据集上超越SOTA,MMEB-V2提升显著。
- 轻量后处理即可实现性能跃升,适合工程部署。
多模态检索依赖单向量检索器,将序列信息压缩为全局表示,但损失了局部关键证据。虽有多种多向量方法,但通常需重新训练且忽略全局摘要。本文提出SMART框架,揭示标准对比学习通过梯度流隐式塑造前序隐藏状态的检索几何。推理时对冻结的隐藏状态进行直接后期交互,可作为即插即用升级,在多种模态上持续提升性能,甚至进一步优化MMEB-V2上的现有SOTA模型。此外,仅需轻量级后训练,既节省计算资源,又在视觉文档检索任务中使单向量模型超越现有多向量方法。SMART提供了高效推理增强与强大微调策略。代码与权重已开源:https://github.com/HanSolo9682/SMART。
原文摘要 · Abstract (English)
Multimodal retrieval relies heavily on single-vector retrievers, which compress rich, sequential token sequences into one single global representation. While efficient, they discard fine-grained, local evidence critical for dense retrieval tasks. Multi-vector approaches were introduced as a solution, but they strictly require training and many ignore the necessity of a globally summarizing representation. To address this, we introduce SMART, a framework that unlocks the latent multi-vector capabilities of standard single-vector models. We first demonstrate that standard contrastive training on the pooled embedding implicitly shapes the retrieval geometry of preceding hidden states via gradient flow. By applying direct late-interaction over these frozen hidden states during inference, SMART acts as a plug-and-play upgrade that consistently improves performance across diverse modalities, improving even the state-of-the-art models further on MMEB-V2. We also reveal SMART's superior performance, as simple lightweight post-training not only saves time and compute, but also brings forth further improvement on Visual Document retrieval, allowing a single-vector model to outperform SoTA multi-vector counterparts. Ultimately, SMART offers both a highly efficient inference enhancement and a powerful finetuning technique for multimodal retrieval. We open source our code and weights at https://github.com/HanSolo9682/SMART.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。