arXiv:2603.14659cs.CVcs.AI2026-03中稿 · ECCV被引 3

用视觉提示训练模型,让视频推理更准更省算力。

VisionCoach: Reinforcing Grounded Video Reasoning via Visual-Perception Prompting

  • 训练时用视觉提示增强关键信息,抑制干扰
  • 在6个基准上达最优,推理无需额外工具
  • 自蒸馏让模型学会不依赖提示也能精准定位

视频推理需在多帧中定位并跟踪与问题相关证据。尽管带可验证奖励的强化学习提升准确率,但在推理过程中仍难以实现可靠的时空定位。现有方法通常依赖大规模标注数据或推理时的感知工具,增加标注或计算成本。为此,我们提出 VisionCoach,一种输入自适应的强化学习框架,通过训练时的视觉提示引导来提升时空定位能力。在强化学习训练中,对挑战性输入选择性应用视觉提示,放大问题相关证据并抑制干扰项。模型通过自蒸馏内化这些改进,从而在推理时直接处理原始视频而无需视觉提示。VisionCoach 包含两个组件:(1) 视觉提示选择器,根据视频和问题预测合适的提示类型;(2) 时空推理器,在视觉提示引导下通过强化学习优化,并使用基于物体意识的定位奖励,确保物体身份一致性和多区域边界框重叠。大量实验表明,VisionCoach 在多个视频推理、理解及时间定位基准(V-STAR、VideoMME、World-Sense、VideoMMMU、PerceptionTest、Charades-STA)上达到当前最优性能,同时保持单一高效推理路径,无需外部工具。结果表明,训练阶段的视觉提示能有效提升定位能力,而自蒸馏使模型在推理时无需提示即可具备该能力。

原文摘要 · Abstract (English)

Video reasoning requires models to locate and track question-relevant evidence across frames. While reinforcement learning (RL) with verifiable rewards improves accuracy, it still struggles to achieve reliable spatio-temporal grounding during the reasoning process. Moreover, improving grounding typically relies on scaled training data or inference-time perception tools, which increases annotation cost or computational cost. To address this challenge, we propose VisonCoach, an input-adaptive RL framework that improves spatio-temporal grounding through visual prompting as training-time guidance. During RL training, visual prompts are selectively applied to challenging inputs to amplify question-relevant evidence and suppress distractors. The model then internalizes these improvements through self-distillation, enabling grounded reasoning directly on raw videos without visual prompting at inference. VisonCoach consists of two components: (1) Visual Prompt Selector, which predicts appropriate prompt types conditioned on the video and question, and (2) Spatio-Temporal Reasoner, optimized with RL under visual prompt guidance and object-aware grounding rewards that enforce object identity consistency and multi-region bounding-box overlap. Extensive experiments demonstrate that VisonCoach achieves state-of-the-art performance under comparable settings, across diverse video reasoning, video understanding, and temporal grounding benchmarks (V-STAR, VideoMME, World-Sense, VideoMMMU, PerceptionTest, and Charades-STA), while maintaining a single efficient inference pathway without external tools. Our results show that visual prompting during training improves grounded video reasoning, while self-distillation enables the model to internalize this ability without requiring prompts at inference time.

视频推理强化学习视觉提示自蒸馏

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。