用标注当最优轨迹,让视频多模态模型强化学习更高效。
Annotations as Rollouts: Efficient and Scalable Reinforcement Learning for Video MLLMs

- 将标注作为直接优化目标,解耦优势估计避免奖励偏差。
- 仅需SFT 2.2倍计算量,推理速度提升36倍,性能全面超越基线。
- 适合追求高效训练与高精度视频理解的开发者和研究者。
多模态大语言模型(MLLM)已成为统一视频感知的主流范式。然而,对大规模多任务数据集进行强化学习后训练仍具挑战性,因现有方法在生成链式思维(CoT)代价高昂的情况下,仍只能采样少量高质量轨迹。本文研究视频MLLM强化学习的样本效率与可扩展性,提出OraRL。我们发现:标注不仅可用于评分轨迹,还可作为直接优化目标的“神谕轨迹”进入策略组。但直接引入神谕轨迹会抬高组基准,导致原本正向的优势反转,称为优势反转。OraRL的核心是解耦优势估计:策略轨迹决定无神谕基线,神谕与策略差距调节方向增益与独立分离的神谕优势。符号平衡剪枝提升效率:仅保留神谕和每组中最强的正负样本,使OraRL计算时间仅为SFT的2.2倍,远低于GRPO-CoT所需的4.9倍。该方法可随模型规模与数据量扩展,从0.8B到9B模型均超越基线,并在10万提示下优于GRPO。无链式思维时,Video-ORA-9B推理仅需130毫秒,而非4,780毫秒。相比先前最佳模型,其在时间mIoU上从62.5提升至66.0,追踪AO从73.0升至78.2,分割指标从64.3升至70.4,三基准空间智能宏平均从51.0升至56.1;在VSI-Bench上达73.1,显著高于GPT-5的55.0与Gemini-3-Pro的55.1。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) have become a prevailing paradigm for unified video perception. However, post-training on large multi-task datasets remains challenging, as existing reinforcement learning methods sample on-policy groups with few high-quality rollouts even with costly chain-of-thought (CoT) generation. In this paper, we study the sample efficiency and scalability of RL post-training for video MLLMs and introduce OraRL. We identify an overlooked role for annotations: Beyond scoring rollouts, each can enter its on-policy group as an oracle rollout, a direct positive optimization target. Direct oracle integration, however, is nontrivial: a high-reward oracle raises the group baseline and inverts otherwise positive policy advantages, a failure we term advantage inversion. At the core of OraRL is a decoupled advantage estimator: policy rollouts determine an oracle-free baseline, while the oracle-policy gap modulates both a directional gain and a separate detached oracle advantage. Sign-balanced pruning improves efficiency: by retaining only the oracle and the strongest rollouts of each sign, OraRL requires just 2.2x the step time of SFT, less than half the 4.9x required by GRPO with CoT. OraRL scales with model size and data, surpassing its backbone from 0.8B to 9B and GRPO up to 100k prompts. Without chain-of-thought, Video-ORA-9B decodes in 130 ms instead of 4,780 ms. Compared with the respective prior best models, it raises temporal mIoU from 62.5 to 66.0, tracking AO from 73.0 to 78.2, segmentation from 64.3 to 70.4, and the three-benchmark spatial-intelligence macro average from 51.0 to 56.1; on VSI-Bench, it scores 73.1 against 55.0 for GPT-5 and 55.1 for Gemini-3-Pro.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。