通过分层自适应推理，让多模态检索更快更省算力。
Skim and Skip: Hierarchical Adaptive Inference for Efficient Multimodal Retrieval
- 先选关键信息 token，再动态决定推理深度
- 保持99%性能，速度提升1.64倍，算力降66.3%
- 适合追求高效多模态检索的开发者与工程师
通用多模态检索（UMR）越来越多采用多模态大语言模型（MLLMs）作为统一嵌入骨干网络，但其强大的检索性能伴随着巨大的推理开销。现有方法通常采用均匀密集推理，即所有输入 token 均通过完整模型，并使用最后一层 [EOS] 表示进行匹配。然而，该范式忽略了多模态检索中的两个关键异质性：不同 token 对最终检索嵌入的贡献高度不均；不同查询所需推理深度差异显著。为此，我们提出 Skim and Skip（SAS），一种用于高效多模态检索的分层自适应推理框架。SAS 首先进行粒度级别的证据选择，仅保留对最终检索嵌入最相关的信息，随后执行深度自适应推理，判断当前表示是否已足够可靠匹配。在 12 个 MMEB 检索任务上的实验表明，SAS 在保持约 99% 密集基线平均检索性能的同时，实现了最高 1.64 倍的端到端加速和最高 66.3% 的 FLOPs 减少。
原文摘要 · Abstract (English)
Universal multimodal retrieval (UMR) increasingly adopts multimodal large language models (MLLMs) as unified embedding backbones, but their strong retrieval performance comes at substantial inference cost. Existing methods typically rely on uniformly dense inference, where all input tokens are processed through the entire model and matched using the final-layer [EOS] representation. However, this paradigm overlooks two key forms of heterogeneity in multimodal retrieval: token contributions to the final retrieval embedding are highly uneven, and different queries require markedly different amounts of inference depth. To address this, we propose Skim and Skip (SAS), a hierarchical adaptive inference framework for efficient multimodal retrieval. SAS first performs token-level evidence selection to preserve only the input information most relevant to the final retrieval embedding, and then performs depth-adaptive inference to determine whether the current representation is already sufficient for reliable matching. Experiments on 12 MMEB retrieval tasks show that SAS retains about 99% of the dense baseline's average retrieval performance while achieving up to 1.64 times end-to-end speedup and up to 66.3% FLOPs reduction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。