通过动态重排轨迹提升多模态大模型强化学习效率
Shuffle-R1: Efficient RL framework for Multimodal Large Language Models via Data-centric Dynamic Shuffle
- 按优势值选取高对比度轨迹，提升梯度质量
- 重排批次中轨迹顺序，让有效样本获得更多训练机会
- 无需额外计算开销，显著提升长程学习效率
强化学习（RL）已成为提升多模态大语言模型（MLLM）推理能力的有效后训练范式。然而，现有RL流程常因两个未被充分关注的问题导致训练效率低下：优势值坍缩（大多数样本的优势集中在零附近），以及回放静音（产生非零梯度的回放比例随时间下降）。这两个问题导致梯度更新次优，阻碍长期学习效率。为此，我们提出Shuffle-R1，一个简单而严谨的框架，通过动态重构轨迹采样与批次构成来提升强化学习微调效率。其核心包括：(1) 成对轨迹采样，选择优势值差异大的高对比轨迹以改善梯度信号质量；(2) 基于优势的轨迹重排，通过有意识地重新组合批次，增加有价值回放的曝光率。在多个推理基准上的实验表明，该框架在几乎无额外开销下持续优于强基线方法。结果凸显了数据驱动适配在高效MLLM强化学习中的重要性。
原文摘要 · Abstract (English)
Reinforcement learning (RL) has emerged as an effective post-training paradigm for enhancing the reasoning capabilities of multimodal large language model (MLLM). However, current RL pipelines often suffer from training inefficiencies caused by two underexplored issues: Advantage Collapsing, where most advantages in a batch concentrate near zero, and Rollout Silencing, where the proportion of rollouts contributing non-zero gradients diminishes over time. These issues lead to suboptimal gradient updates and hinder long-term learning efficiency. To address these issues, we propose Shuffle-R1, a simple yet principled framework that improves RL fine-tuning efficiency by dynamically restructuring trajectory sampling and batch composition. It introduces (1) Pairwise Trajectory Sampling, which selects high-contrast trajectories with large advantages to improve gradient signal quality, and (2) Advantage-based Trajectory Shuffle, which increases exposure of valuable rollouts through informed batch reshuffling. Experiments across multiple reasoning benchmarks show that our framework consistently outperforms strong RL baselines with minimal overhead. These results highlight the importance of data-centric adaptations for more efficient RL training in MLLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。