让视频多模态模型自动调节推理长度,更聪明地分配思考时间。
CARE: Competence-Aware Reward Shaping for Adaptive Reasoning Length in Video-MLLMs

- 用平滑的通过率估计模型能力,动态调整奖励策略。
- 训练中推理长度先增后减,最终更短但信息量更高。
- 无需额外计算开销,适合视频理解与多模态推理任务。
在多模态视频推理中,基于强化学习的方法通常采用简单且僵化的推理长度控制策略,无法随模型能力演变自适应调整。这种不匹配可能在早期抑制必要探索,而在模型变强后导致冗余推理和低效解码。本文提出CARE,一种面向多模态推理的、具备能力感知的奖励重塑框架,用于自适应优化推理长度。CARE通过指数移动平均维护平滑的能力估计,并据此将训练过程划分为渐进阶段,逐步将奖励偏好从鼓励长篇推理转向追求简洁高效。为避免将冗长表述误认为任务复杂,CARE引入批次级统计进行推理努力归一化,并设计后验放大器,增强对历史难点样本上表现异常优异情况的奖励信号。该机制无缝集成于GRPO训练流程,无额外推理开销。在多个视频推理与通用视频理解基准上的大量实验表明,CARE显著提升推理准确率,稳定强化学习过程,并大幅提高令牌效率。此外,其展现出训练期间推理长度呈倒U型变化轨迹,收敛时生成更短但更具信息量的推理链,证明了推理预算的有效分配。源代码已公开于https://github.com/1Pansy/Video-CARE。
原文摘要 · Abstract (English)
In multimodal video reasoning, reinforcement learning-based methods typically rely on simplistic and inflexible reasoning-length control strategies that fail to adapt to the model's evolving competence. This mismatch may suppress necessary exploration at early stages, while encouraging redundant reasoning and inefficient decoding once the model becomes more competent. In this paper, we propose CARE, a competence-aware reward shaping framework for adaptive reasoning length optimization in multimodal reasoning. Specifically, CARE maintains a smoothed competence estimate via an exponential moving average of pass rates, and uses it to route training into progressive stages that shift the reward preference from exploration-oriented long-form reasoning to efficiency-oriented concise reasoning. To avoid conflating verbosity with intrinsic task complexity, CARE further normalizes reasoning effort with batch-level statistics, and introduces a posterior amplifier to strengthen reward signals for unexpectedly strong performance on historically difficult samples. The proposed mechanism is seamlessly integrated into the GRPO training pipeline and incurs no additional inference-time overhead. Extensive experiments on multiple video reasoning and general video understanding benchmarks demonstrate that CARE consistently improves reasoning accuracy, stabilizes reinforcement learning, and significantly enhances token efficiency. Moreover, CARE exhibits a characteristic inverted-U trajectory of reasoning length during training, and yields shorter yet more informative reasoning traces at convergence, indicating effective adaptive allocation of reasoning budget. We provide the source code for our proposed CARE framework and experiments at https://github.com/1Pansy/Video-CARE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。