让视频检索理解编辑后的隐含变化,提升精准度与可解释性。
CoVR-R:Reason-Aware Composed Video Retrieval
- 基于大模型推理编辑的因果与时间后效,生成更准确的查询
- 零样本下在隐含效果任务上显著超越基线,召回率更高
- 适合需要理解复杂视觉变化的视频搜索场景
组成式视频检索(CoVR)旨在给定参考视频和文本修改指令时找到目标视频。现有方法假设修改文本完全描述视觉变化,忽略了编辑带来的隐含后效(如运动、状态转移、视角或持续时间变化)。本文提出一种以推理为核心的零样本方法,利用大型多模态模型推断编辑所隐含的因果与时间后效,并将推理生成的查询与候选视频对齐,无需任务特定微调。为评估推理能力,我们构建了CoVR-Reason基准,包含结构化推理轨迹和需预测后效的挑战性干扰项。实验表明,该方法在召回率@K上优于强基线,尤其在隐含效应子集上表现突出。自动与人工分析证实结果具有更高的步骤一致性与效应真实性。研究证明,将推理融入通用多模态模型可有效实现CoVR,通过显式建模因果与时间后效,减少对特定标注数据的依赖,提升对复杂隐含变化的泛化能力与检索结果的可解释性。模型、代码与基准已开源。
原文摘要 · Abstract (English)
Composed Video Retrieval (CoVR) aims to find a target video given a reference video and a textual modification. Prior work assumes the modification text fully specifies the visual changes, overlooking after-effects and implicit consequences (e.g., motion, state transitions, viewpoint or duration cues) that emerge from the edit. We argue that successful CoVR requires reasoning about these after-effects. We introduce a reasoning-first, zero-shot approach that leverages large multimodal models to (i) infer causal and temporal consequences implied by the edit, and (ii) align the resulting reasoned queries to candidate videos without task-specific finetuning. To evaluate reasoning in CoVR, we also propose CoVR-Reason, a benchmark that pairs each (reference, edit, target) triplet with structured internal reasoning traces and challenging distractors that require predicting after-effects rather than keyword matching. Experiments show that our zero-shot method outperforms strong retrieval baselines on recall at K and particularly excels on implicit-effect subsets. Our automatic and human analysis confirm higher step consistency and effect factuality in our retrieved results. Our findings show that incorporating reasoning into general-purpose multimodal models enables effective CoVR by explicitly accounting for causal and temporal after-effects. This reduces dependence on task-specific supervision, improves generalization to challenging implicit-effect cases, and enhances interpretability of retrieval outcomes. These results point toward a scalable and principled framework for explainable video search. The model, code, and benchmark are available at https://github.com/mbzuai-oryx/CoVR-R.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。