用视觉语言模型零样本融合多路检索结果，提升视频检索精度。
Vote-in-Context: Turning VLMs into Zero-Shot Rank Fusers
- 将检索结果与元信息拼入提示，让VLM基于内容判断优劣
- 在VATEX数据集上实现99.6%的v2t Recall@1，提升超40个百分点
- 无需训练，适合快速部署到复杂多模态检索场景
在检索领域，从异构检索器中融合候选结果是长期挑战，尤其针对视频等复杂多模态数据。现有无训练融合方法仅依赖排序或得分信号，忽略候选内容表示。本文提出Vote-in-Context（ViC），一种通用的无训练框架，将列表级重排序与融合视为视觉语言模型（VLM）的零样本推理任务。核心思想是将内容证据和检索器元信息直接序列化至VLM提示中，使模型能自适应权衡检索器共识与视觉-语言内容。为处理跨模态视频检索，引入S-Grid：一种紧凑序列化映射，将每个视频表示为图像网格，可选搭配字幕以支持列表级推理。ViC在单列表重排序和集成融合两种场景均表现优异，在ActivityNet、VATEX等视频检索基准上达到新最优零样本性能。在MSR-VTT上实现87.1%（t2v）/89.0%（v2t）Recall@1，VATEX上达99.6%（v2t），相比前代基线提升高达+40 Recall@1。代码与资源已公开。
原文摘要 · Abstract (English)
In the retrieval domain, candidates' fusion from heterogeneous retrievers is a long-standing challenge, particularly for complex, multi-modal data such as videos. While typical fusion techniques are training-free, they rely solely on rank or score signals, disregarding candidates' representations. This work introduces Vote-in-Context (ViC), a generalized, training-free framework that re-thinks list-wise reranking and fusion as a zero-shot reasoning task for a Vision-Language Model (VLM). The core insight is to serialize both content evidence and retriever metadata directly within the VLM's prompt, allowing the model to adaptively weigh retriever consensus against visual-linguistic content. We demonstrate the generality of this framework by applying it to the challenging domain of cross-modal video retrieval. To this end, we introduce the S-Grid, a compact serialization map that represents each video as an image grid, optionally paired with subtitles to enable list-wise reasoning over video candidates. ViC is evaluated both as a single-list reranker, where it dramatically improves the precision of individual retrievers, and as an ensemble fuser, where it consistently outperforms strong baselines like CombSUM. Across video retrieval benchmarks including ActivityNet and VATEX, the framework establishes new state-of-the-art zero-shot retrieval performance, demonstrating its effectiveness in handling complex visual and temporal signals alongside text. In zero-shot settings, ViC achieves Recall@1 scores of 87.1% (t2v) / 89.0% (v2t) on MSR-VTT and 99.6% (v2t) on VATEX, representing massive gains of up to +40 Recall@1 over previous state-of-the-art baselines. We present ViC as a simple, reproducible, and highly effective recipe for turning modern VLMs into powerful zero-shot rerankers and fusers. Code and resources are publicly available at: https://github.com/mohammad2012191/ViC
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。