让AI像人一样操作视频来推理，提升复杂视频理解能力。
Video-CoM: Interactive Video Reasoning via Chain of Manipulations
- 通过可交互的视觉操作链进行多步推理，动态获取证据。
- 在9个基准上平均提升3.6%，仅用2.5万条监督数据训练。
- 适合需要精细时空分析的视频理解任务，如动作预测与验证。
近期多模态大模型虽提升了视频理解能力，但大多在视频编码后仅以文本形式进行推理，将视觉输入视为静态上下文。这种被动模式导致语义瓶颈：模型无法重看、重聚焦或验证证据，难以实现细粒度时空理解。本文提出交互式视频推理新范式，将视频变为动态认知工作区，使模型能“与视频共思”。所提出的 Video CoM 模型通过链式操作（Chain of Manipulations, CoM）执行迭代视觉操作，持续收集并优化证据。为支持该行为，构建了包含18,000条指令的 Video CoM Instruct 数据集，用于多步操作推理。除监督学习外，还采用感知推理的分组相对策略优化（GRPO）进行强化学习训练。相比仅依赖稀疏答案奖励的方法，本方法引入步骤级推理奖励，引导模型生成更可靠、一致的推理过程。Video CoM 在九个视频推理基准上表现优异，平均性能优于最新模型3.6%，且仅需2.5万条SFT和3千条GRPO视频样本训练，远低于同类大规模模型。消融实验表明，感知推理奖励同时提升准确率与可解释性。
原文摘要 · Abstract (English)
Recent multimodal large language models (MLLMs) have advanced video understanding, yet most still "think about videos" ie once a video is encoded, reasoning unfolds entirely in text, treating visual input as a static context. This passive paradigm creates a semantic bottleneck: models cannot rewatch, refocus, or verify evidence, leading to shallow visual reasoning on tasks requiring fine grained spatio temporal understanding. In this work, we introduce Interactive Video Reasoning, a new paradigm that transforms video into an active cognitive workspace, enabling models to "think with videos". Our model, Video CoM, reasons through a Chain of Manipulations (CoM), performing iterative visual actions to gather and refine evidence. To support this behavior, we construct Video CoM Instruct, an 18K instruction tuning dataset curated for multi step manipulation reasoning. Beyond supervised learning, we further optimize the manipulation policy via reinforcement learning with reasoning aware Group Relative Policy Optimization (GRPO). Unlike prior work that relies solely on sparse answer rewards, our method introduces step level reasoning rewards, guiding the model toward grounded and consistent reasoning. Video CoM achieves strong results across nine video reasoning benchmarks, improving average performance by 3.6 percent over recent state of the art models, while training on only 25K SFT and 3K GRPO video samples, significantly fewer than comparable large scale models. Ablation studies demonstrate that reasoning aware rewards improve both accuracy and interpretability. Code: https://github.com/mbzuai-oryx/Video-CoM
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。