让大模型多层多槽提取视频文本匹配信号,提升细粒度检索效果。
MARS: What Retrieval Signals Are Hidden in Multimodal Large Language Models for Text-Video Retrieval?

- 用多层隐藏状态构建多个可适应的表示槽,融合多层次信息。
- 在4个数据集上达到最新最优,尤其在难例重排任务中提升显著。
- 适合需要精准视频检索的研究者和工业应用开发者。
文本-视频检索需区分场景、动作和时间模式相似的视频。当前多模态大模型常将输入压缩为单个最终层令牌,易丢失多样化视频-文本线索,限制细粒度检索。为此,本文提出MARS:一种多层多槽嵌入框架。MARS通过融合不同解码器层的隐藏状态,构建多个自适应表示槽,对齐文本与视频槽并聚合其相似性以实现检索。为进一步应对混淆候选,引入硬负样本感知的槽特化目标,促使各槽捕捉判别性匹配线索。在四个文本-视频检索基准测试中,MARS在直接相似性检索与重排序设置下均取得领先性能。消融实验与分析表明,多层融合、多槽设计与硬负样本感知特化带来互补增益。代码已开源。
原文摘要 · Abstract (English)
Text-video retrieval requires representations that can distinguish videos with similar scenes, actions, and temporal patterns. Recent multimodal large language models have been adapted as embedding models, but they often represent each input using a single token from the final layer. This can compress diverse video-text cues into a single vector and limit fine-grained retrieval. To address this limitation, we propose MARS, a multi-layer and multi-slot embedding framework for text-video retrieval. MARS constructs multiple adaptive representation slots by combining hidden states from different decoder layers, compares corresponding text and video slots, and aggregates their similarities for retrieval. To better handle confusing candidates, we further introduce a hard-negative-aware slot specialization objective that encourages the slots to capture discriminative matching cues. Experiments on four text-video retrieval benchmarks show that MARS achieves state-of-the-art results in both direct similarity-based retrieval and reranking settings. Ablation studies and analyses demonstrate that multi-layer fusion, multiple slots, and hard-negative-aware slot specialization provide complementary gains. Code is available at https://github.com/sejong-rcv/MARS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。