用框推理提升视频定位精度,让大模型更懂时空位置。
Thinking With Bounding Boxes: Enhancing Spatio-Temporal Video Grounding via Reinforcement Fine-Tuning
- 引入框式思维链,中间步骤显式推理时空位置
- 在HCSTVG-v1上比最优方法高7.3% m_tIoU
- 无需改架构,适合作为通用视频定位基础模型
时空视频定位(STVG)需从自然语言描述中精准定位未剪辑视频中的目标对象在时间和空间上的位置。尽管多模态大语言模型(MLLMs)具备强大的语言理解能力,但在STVG任务上表现不佳,主要因训练目标不匹配及视觉编码器细粒度区域-词对齐能力弱。为此,我们提出STVG-o1,首个无需任何结构修改即可使现成MLLM达到顶尖性能的框架。该方法引入框式思维链机制,在生成最终预测前显式推理时空位置;并设计包含格式、一致性、时间、空间和思考奖励的多维强化奖励函数,通过强化微调提供几何感知监督。在HCSTVG-v1/v2和VidSTG数据集上评估,STVG-o1在HCSTVG上刷新纪录,较最佳专用方法提升7.3% m_tIoU,与专用模型持平于VidSTG,显著超越所有现有基于MLLM的方法。其在跨数据集上展现出强开放词汇泛化能力,确立了MLLM作为精确时空定位强大骨干的可行性。代码与模型将公开。
原文摘要 · Abstract (English)
Spatio-temporal video grounding (STVG) requires localizing a target object in untrimmed videos both temporally and spatially from natural language descriptions. Despite their strong language understanding, multimodal large language models (MLLMs) underperform on STVG due to misaligned training objectives and weak fine-grained region-word alignment in standard visual encoders. To address this, we propose STVG-o1, the first framework that enables off-the-shelf MLLMs to achieve state-of-the-art STVG performance without any architectural modifications. Our method introduces a bounding-box chain-of-thought mechanism that explicitly reasons about spatio-temporal locations in an intermediate step before producing the final prediction. We further design a multi-dimensional reinforcement reward function consisting of format, consistency, temporal, spatial, and think rewards, which provides geometry-aware supervision through reinforcement fine-tuning. Evaluated on HCSTVG-v1/v2 and VidSTG, STVG-o1 sets new state-of-the-art results on HCSTVG, outperforming the best task-specific method by 7.3\% m\_tIoU on HCSTVG-v1, matching specialized models on VidSTG, and surpassing all existing MLLM-based approaches by large margins. It also demonstrates strong open-vocabulary generalization across datasets, establishing MLLMs as viable and powerful backbones for precise spatio-temporal grounding. Our code and models will be released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。