通过视觉缓存复用与稀疏交互，实现高效多模态重排序。
miniReranker: Efficient Multimodal Reranking through Visual Cache Reuse and Interaction Sparsity
- 提出视觉优先的重排序框架，提升缓存复用效率。
- 在高复用场景下，推理时间低于密集模型1%，性能保留96%以上。
- 通过早期退出、注意力限制和嵌入引导剪枝，显著降低计算开销。
多模态大语言模型（MLLM）最近展现出作为点式重排序器的强大潜力，通过下一词预测直接建模查询-文档相关性。然而，点式重排序在查询-文档对间存在大量重复计算，而变压器的因果结构仅允许通过预缓存复用前缀片段。为解决现有查询优先与文档优先格式在视觉问答提示风格与计算感知复用之间的不匹配问题，我们提出一种‘视觉优先’范式，同时提升缓存复用效率与重排序性能。但剩余开销仍较大，主要来自三方面：(1) 模型深度，通过早期退出减少活跃参数；(2) 跨段注意力，将其限制在少数层内的窄交互带；(3) 视觉标记，通过嵌入引导剪枝减少标记数量。上述设计共同构成miniReranker，使得单个查询在高复用设置下的重排序运行时间低于密集实现的1%，同时保持超过96%的密集模型性能。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) have recently shown strong potential as point-wise rerankers by directly modeling query--document relevance through next-token prediction. However, point-wise reranking suffers from substantial repeated computation across query--document pairs, while the causal structure of transformers allows only prefix segments to be reused via pre-caching. To address the misalignment of existing query-first and document-first formats with both VQA-style prompting and computation-aware reuse, we propose a $\textit{vision-first}$ formulation that improves both cache reuse efficiency and reranking performance. However, the remaining cost is still considerable and stems from three main sources: (1) $\textit{model depth}$, for which we reduce active parameters via early exit; (2) $\textit{cross-segment attention}$, which we restrict to a narrow interaction band across a few layers; and (3) $\textit{visual tokens}$, where we reduce the number of tokens via embedder-guided pruning. Together, these designs form miniReranker, which reduces reranking runtime to <1% of the dense implementation under high-reuse settings for a single query, while preserving >96% of the dense model performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。