通过连续空间优化查询,实现多轮视频检索与精准定位。
VideoSearch-R1: Iterative Video Retrieval and Reasoning via Soft Query Refinement

- 在连续潜空间中软化调整查询,避免文本重写
- 在三个数据集上达成最新最优,支持迭代检索与定位
- 适合需要精确视频内容定位的智能搜索场景
随着视频数据规模与任务复杂度持续增长,亟需能从大规模视频库中检索相关视频(跨视频推理),并在检索结果内执行细粒度、查询相关的任务(如时间定位,即视频内推理)。现有方法通常将检索视为预处理步骤,一旦初始检索失败,缺乏修正机制,导致后续推理失败。尽管近期代理框架推动了视频理解进展,但大多假设目标视频已知,仅聚焦于视频内推理。为此,本文提出 VideoSearch-R1,一种基于多轮交互的代理框架,实现迭代式视频检索与推理。核心是引入软查询精炼(SQR),在连续潜空间中调整查询词,而非离散文本重写,提升调整效率与精度。SQR与推理过程通过组相对策略优化(GRPO)训练,利用检索与下游任务的任务级奖励信号进行引导。实验表明,VideoSearch-R1 在三个数据集上的视频语境时刻检索(VCMR)任务中达到当前最佳性能,可迭代检索大尺度视频库,精细优化查询,并在检索内容内完成精准的时间定位。分析显示,SQR能有效改进原始查询,生成所需词数显著少于显式的文本重写方式。代码与模型权重已公开于 mlvlab.github.io/VideoSearch-R1。
原文摘要 · Abstract (English)
As video corpora continue to expand in both scale and task complexity, there is increasing demand for approaches that retrieve relevant videos from large-scale corpora (inter-video reasoning) and subsequently perform fine-grained, query-conditioned tasks (intra-video reasoning) within the retrieved content, such as temporal grounding. However, existing approaches typically treat retrieval as a preprocessing step, and consequently, when the initial retrieval fails, there is no mechanism to refine the search, leading to the failure of subsequent fine-grained intra-video reasoning. Moreover, while recent agentic frameworks have advanced video understanding, they typically assume that the query-relevant video is already given, focusing exclusively on intra-video reasoning tasks. To address these limitations, we propose VideoSearch-R1, an agentic framework for iterative video retrieval and reasoning through multi-turn interaction with a video search engine. Specifically, we introduce Soft Query Refinement (SQR) to refine search query tokens in a continuous latent space rather than rewriting queries in the discrete text space, enabling more efficient and fine-grained adjustments. SQR and its reasoning process are trained using Group Relative Policy Optimization (GRPO), guided by task-level reward signals derived from retrieval and downstream tasks. Building upon this, VideoSearch-R1 achieves state-of-the-art performance across three datasets on Video Corpus Moment Retrieval (VCMR), iteratively retrieving videos from large-scale corpora, refining search queries, and performing precise query-conditioned temporal grounding within the retrieved content. Our analyses show that SQR effectively refines the original query, requiring significantly fewer generated tokens than explicit text-level query refinement. Code and model checkpoints are publicly available at mlvlab.github.io/VideoSearch-R1.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。