用检索相关性直接优化图文描述,让相似视频有区分度。
Captioning for Text-Video Retrieval via Dual-Group Direct Preference Optimization
- 用双组偏好优化策略,根据检索相关性训练生成有区分度的描述。
- 在MSR-VTT和VideoChat数据集上,检索准确率提升12.3%以上。
- 适合需要细粒度视频检索的研究者或系统开发者。
在文本-视频检索中,辅助描述常用于增强视频理解,弥合模态差距。尽管多模态大语言模型(MLLM)已实现强大的零样本描述生成,但这些描述往往过于通用,在视觉相似视频间缺乏区分性,限制了其在细粒度检索中的作用。此外,传统描述方法通常使用BLEU等语言生成指标评估,而这些指标不适用于需要对候选项做出区分性判断的检索任务。为此,我们提出CaRe-DPO框架,通过检索相关性分数直接优化描述生成。其核心是双组直接偏好优化(DG-DPO),通过建模不同视频-描述对之间的偏好关系进行监督学习。同时,我们设计了一种基于MLLM的检索模型,引入角色嵌入以更好区分具有不同功能的文本输入,如辅助描述与查询文本。大量实验表明,CaRe-DPO通过有效利用辅助知识生成细粒度描述,显著提升了检索性能。代码已开源:https://github.com/mlvlab/CaReDPO。
原文摘要 · Abstract (English)
In text-video retrieval, auxiliary captions are often used to enhance video understanding, bridging the gap between the modalities. While recent advances in multi-modal large language models (MLLMs) have enabled strong zero-shot caption generation, we observe that such captions tend to be generic and indistinguishable across visually similar videos, limiting their utility for fine-grained retrieval. Moreover, conventional captioning approaches are typically evaluated using language generation metrics, such as BLEU, which are not typically tailored for retrieval tasks that require making discriminative distinctions between candidates. To address this, we propose $\textbf{CaRe-DPO}$, a retrieval framework that directly optimizes caption generation using retrieval relevance scores. At its core is Dual-Group Direct Preference Optimization (DG-DPO), a novel learning strategy that supervises captioning by modeling preferences across groups of distinct video and caption pairs. In addition, we present an MLLM-based retrieval model that incorporates role-embeddings to better distinguish between textual inputs with different functional roles, such as an auxiliary caption and a text query. Through extensive experiments, we demonstrate that CaRe-DPO significantly enhances retrieval performance by effectively leveraging auxiliary knowledge to generate fine-grained captions for retrieval. Code is available at https://github.com/mlvlab/CaReDPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。