通过思维链机制提升视频理解与分割的时序推理能力。
RCoT-Seg: Reinforced Chain-of-Thought for Video Reasoning and Segmentation

- 分两阶段处理:先用智能体选择关键帧,再进行高精度分割。
- 在多个数据集上优于现有方法,关键帧选择更准确、跨帧一致性更强。
- 适合需要精准时序理解的复杂多对象视频分析任务。
视频推理分割(VRS)旨在根据隐含指令(表达人类意图与时间逻辑)对视频中的目标物体进行分割。现有基于多模态大模型的方法通常通过简单采样或辅助模型选择帧后,以[SEG]标记生成掩码,但受限于监督信号不足和帧-语言相似性规则,导致关键帧选择范围狭窄,削弱了全局时序理解,在复杂多对象场景中定位易出错。为此,我们提出RCoT-Seg,一种视频思维链框架,将VRS分解为时序视频推理(TVR)与关键帧目标感知(KTP)两个阶段,显式分离时序推理与空间感知。在TVR阶段,设计了一个智能体式关键帧选择模块,基于精心构建的思维链起始语料库初始化,并通过任务对齐奖励下的GRPO算法迭代优化,实现自我评估下的生成与重选,增强时刻定位与时序推理能力。在KTP阶段,RCoT-Seg在选定帧上执行高分辨率分割,并采用SAM2-based方法传播掩码至全序列,替代启发式采样与外部选择器,显著提升空间精度与帧间一致性。大量实验表明,所提RCoT-Seg在多个基准上均优于当前最先进方法。代码与模型将公开于https://github.com/Victor-wjw/RCoT-Seg。
原文摘要 · Abstract (English)
Video Reasoning Segmentation (VRS) aims to segment target objects in videos based on implicit instructions that convey human intent and temporal logic. Existing MLLM-based methods predict masks with a [SEG] token after selecting frames via simple sampling or an auxiliary MLLM, where limited supervision and frame-language similarity rules often yield narrow-scope keyframe choices that weaken holistic temporal understanding and lead to brittle localization in complex multi-object scenes. To address these issues, we introduce RCoT-Seg, a video-of-thought framework that factorizes VRS into temporal video reasoning (TVR) and keyframe target perception (KTP), explicitly separating temporal reasoning from spatial perception. Specifically, in the TVR stage, an agentic keyframe selection module, initialized with a curated CoT-start corpus and refined by GRPO under task-aligned rewards, is proposed to generate and reselect the keyframe through self-evaluation, strengthening moment localization and temporal reasoning. In the KTP stage, RCoT-Seg performs high-resolution segmentation on the selected frame and propagates masks with SAM2-based methods across the sequence, replacing heuristic sampling and external selectors while improving spatial precision and inter-frame consistency. Extensive experimental results demonstrate that the proposed RCoT-Seg achieves favorable performance against the state-of-the-art methods. The code and models will be publicly released at https://github.com/Victor-wjw/RCoT-Seg.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。