arXiv:2601.16582cs.CV2026-01

用跨模态对齐模块提升视频检索精度,零样本泛化能力强。

X-Aligner: Composed Visual Retrieval without the Bells and Whistles

  • 引入X-Aligner渐进融合视觉与文本特征,对齐目标视频表征。
  • 在Webvid-CoVR测试集上达到63.93%的Recall@1,领先当前水平。
  • 适用于需要高精度图文检索的场景,尤其适合零样本迁移任务。

组合式视频检索(CoVR)通过结合视觉与文本查询实现精准检索。现有框架多采用单阶段融合多模态输入,性能提升有限。本文提出新框架,利用视觉语言模型(VLM)的表征能力,引入X-Aligner跨注意力模块,逐步融合并对齐视觉与文本特征,使其与目标视频表征一致。为增强多模态查询表示,额外引入视觉查询的标题作为输入。框架分两阶段训练:第一阶段仅训练新增模块,第二阶段微调文本查询编码器。基于BLIP与BLIP-2架构,在Webvid-CoVR数据集上进行训练。除在Webvid-CoVR-Test上评估外,还进行了零样本评估,覆盖组合图像检索(CIR)数据集CIRCO和Fashion-IQ。结果表明,该框架在CoVR任务中达到当前最优,测试集上Recall@1达63.93%,并在CIR任务上展现出强零样本泛化能力。

原文摘要 · Abstract (English)

Composed Video Retrieval (CoVR) facilitates video retrieval by combining visual and textual queries. However, existing CoVR frameworks typically fuse multimodal inputs in a single stage, achieving only marginal gains over initial baseline. To address this, we propose a novel CoVR framework that leverages the representational power of Vision Language Models (VLMs). Our framework incorporates a novel cross-attention module X-Aligner, composed of cross-attention layers that progressively fuse visual and textual inputs and align their multimodal representation with that of the target video. To further enhance the representation of the multimodal query, we incorporate the caption of the visual query as an additional input. The framework is trained in two stages to preserve the pretrained VLM representation. In the first stage, only the newly introduced module is trained, while in the second stage, the textual query encoder is also fine-tuned. We implement our framework on top of BLIP-family architecture, namely BLIP and BLIP-2, and train it on the Webvid-CoVR data set. In addition to in-domain evaluation on Webvid-CoVR-Test, we perform zero-shot evaluations on the Composed Image Retrieval (CIR) data sets CIRCO and Fashion-IQ. Our framework achieves state-of-the-art performance on CoVR obtaining a Recall@1 of 63.93% on Webvid-CoVR-Test, and demonstrates strong zero-shot generalization on CIR tasks.

视频检索多模态跨模态对齐

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。