AERO让强化学习更省算力，自动淘汰无效生成，提升训练效率。
Train Less, Learn More: Adaptive Efficient Rollout Optimization for Group-Based Reinforcement Learning
- 根据生成结果动态调整滚动生成数量，避免无效计算。
- 在相同预算下减少48%训练算力，每步提速45%以上。
- 适合追求高效对齐大模型的开发者与研究者使用。
强化学习在大语言模型后训练中扮演核心角色。现有方法中，组相对策略优化（GRPO）广泛用于可验证奖励的强化学习微调（RLVR）。GRPO固定每轮生成规模为 $N$，当组内所有样本结果一致时，优势值归零，导致梯度消失，浪费计算资源。本文提出自适应高效滚动生成优化（AERO），通过自适应策略、选择性剔除机制及贝叶斯后验维护，有效避免零优势死区。在 Qwen2.5-Math-1.5B、Qwen2.5-7B 及 Qwen2.5-7B-Instruct 三种模型配置上，AERO在不牺牲性能的前提下，显著提升效率：相同总滚动生成预算下，训练算力减少约48%，单步壁钟时间缩短约45%。尽管算力大幅降低，其 Pass@8 和 Avg@8 指标仍与或优于 GRPO，展现出一种实用、可扩展且高效的基于强化学习的大模型对齐策略。
原文摘要 · Abstract (English)
Reinforcement learning (RL) plays a central role in large language model (LLM) post-training. Among existing approaches, Group Relative Policy Optimization (GRPO) is widely used, especially for RL with verifiable rewards (RLVR) fine-tuning. In GRPO, each query prompts the LLM to generate a group of rollouts with a fixed group size $N$. When all rollouts in a group share the same outcome, either all correct or all incorrect, the group-normalized advantages become zero, yielding no gradient signal and wasting fine-tuning compute. We introduce Adaptive Efficient Rollout Optimization (AERO), an enhancement of GRPO. AERO uses an adaptive rollout strategy, applies selective rejection to strategically prune rollouts, and maintains a Bayesian posterior to prevent zero-advantage dead zones. Across three model configurations (Qwen2.5-Math-1.5B, Qwen2.5-7B, and Qwen2.5-7B-Instruct), AERO improves compute efficiency without sacrificing performance. Under the same total rollout budget, AERO reduces total training compute by about 48% while shortening wall-clock time per step by about 45% on average. Despite the substantial reduction in compute, AERO matches or improves Pass@8 and Avg@8 over GRPO, demonstrating a practical, scalable, and compute-efficient strategy for RL-based LLM alignment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。