让AI像人一样分步推理视频目标分割,提升复杂场景下的定位精度。
VideoSEG-O3: A Multi-turn Reinforcement Learning Framework for Reasoning Video Object Segmentation

- 采用多轮强化学习模拟人类从粗到细的思考过程
- 通过迭代定位关键帧和时间段,实现像素级精准分割
- 设计专用数据集支持复杂推理,适合视频理解研究者
推理式视频目标分割(RVOS)需要融合时序动态、空间细节与语言推理以实现精确的像素级定位。现有方法仅依赖固定初始输入,缺乏主动获取视觉证据的能力,难以处理长视频或复杂引用。为此,我们提出首个多轮强化学习框架 VideoSEG-O3,模拟人类‘由粗到细’的认知过程。该框架采用多轮时空思维链,通过迭代定位关键时间段与关键帧捕捉细粒度信息。为使策略在强化学习阶段感知分割质量,而非仅依赖[SEG]文本概率,我们引入分割感知的逻辑校准机制,将像素级分割反馈直接融入词元级逻辑值。此外,我们设计解耦的思维轨迹,将推理过程分层分解为时序、空间与语言维度,并构建了专用于冷启动的 VTS-CoT 数据集,包含全面的推理轨迹。代码与模型将公开于 https://github.com/Dmmm1997/VideoSEG-O3。
原文摘要 · Abstract (English)
Reasoning Video Object Segmentation (RVOS) demands a sophisticated integration of temporal dynamics, spatial details, and linguistic reasoning to achieve precise pixel-level localization. Existing methods are limited to reasoning over fixed initial inputs and lack the capacity to actively acquire further visual evidence, which is often essential for resolving complex references in long or intricate videos. To address this, we propose \textbf{VideoSEG-O3}, the first multi-turn reinforcement learning framework for RVOS that emulates the human \textit{``coarse-to-fine''} cognitive process. It employs a \textit{multi-turn temporal-spatial chain-of-thought} to capture fine-grained details by iteratively pinpointing critical intervals and keyframes. Additionally, to enable the policy to perceive segmentation quality beyond mere text probability of \texttt{[SEG]} during the RL stage, we introduce \textit{SEG-aware logit calibration}, which integrates pixel-wise segmentation feedback directly into the token-level logits. Furthermore, we design a \textit{decoupled thinking trace} to hierarchically decompose the reasoning process into temporal, spatial, and linguistic dimensions, and construct \textbf{VTS-CoT}, a specialized cold-start dataset featuring comprehensive reasoning trajectories. The code and models will be released at https://github.com/Dmmm1997/VideoSEG-O3.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。