用强化学习让视频模型更准地定位物体,不依赖额外标注。
STVG-R1: Incentivizing Instance-Level Reasoning and Grounding in Videos via Reinforcement Learning
- 给每个物体分配唯一编号作为视觉提示,避免图文坐标错位。
- 在六大数据集上提升20.9%的定位准确率,刷新视频定位新纪录。
- 零样本迁移能力强,可直接用于多对象视频分割任务。
在视觉语言模型中,文本描述与视觉坐标的错位常导致幻觉问题,尤其在密集预测任务如时空视频定位(STVG)中更为严重。以往方法多聚焦于增强图文对齐或引入辅助解码器,但需额外可训练模块,带来高标注成本和计算开销。本文提出一种新型视觉提示范式,将每帧坐标预测转化为紧凑的实例级识别问题:为每个物体分配唯一且时间一致的ID,并将其嵌入视频作为视觉提示,提供明确可解释的输入。进一步提出STVG-R1,首个面向STVG的强化学习框架,通过任务驱动奖励联合优化时间准确性、空间一致性与结构格式正则化。在六个基准上的实验表明,该方法显著优于基线Qwen2.5-VL-7B,在HCSTVG-v2上以20.9%的m_IoU提升达到新SOTA。令人意外的是,其零样本泛化能力极强,可在多对象指代视频分割任务上达到47.3%的J&F,同样为当前最优。
原文摘要 · Abstract (English)
In vision-language models (VLMs), misalignment between textual descriptions and visual coordinates often induces hallucinations. This issue becomes particularly severe in dense prediction tasks such as spatial-temporal video grounding (STVG). Prior approaches typically focus on enhancing visual-textual alignment or attaching auxiliary decoders. However, these strategies inevitably introduce additional trainable modules, leading to significant annotation costs and computational overhead. In this work, we propose a novel visual prompting paradigm that avoids the difficult problem of aligning coordinates across modalities. Specifically, we reformulate per-frame coordinate prediction as a compact instance-level identification problem by assigning each object a unique, temporally consistent ID. These IDs are embedded into the video as visual prompts, providing explicit and interpretable inputs to the VLMs. Furthermore, we introduce STVG-R1, the first reinforcement learning framework for STVG, which employs a task-driven reward to jointly optimize temporal accuracy, spatial consistency, and structural format regularization. Extensive experiments on six benchmarks demonstrate the effectiveness of our approach. STVG-R1 surpasses the baseline Qwen2.5-VL-7B by a remarkable margin of 20.9% on m_IoU on the HCSTVG-v2 benchmark, establishing a new state of the art (SOTA). Surprisingly, STVG-R1 also exhibits strong zero-shot generalization to multi-object referring video object segmentation tasks, achieving a SOTA 47.3% J&F on MeViS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。