大批次能否加速LLM强化学习,取决于吞吐提升是否超过样本消耗增加。
When Do Larger Batches Help Scale LLM Reinforcement Learning?

- 通过固定总样本量比较不同批次,发现批量大小对学习轨迹影响有限。
- 在相同硬件下,大批次可使生成吞吐提升最多2.29倍。
- 仅当吞吐增益超过样本成本时,大批次才能缩短训练时间,适合系统优化者。
更大的批次能降低每次更新的随机梯度方差,理论上应加速训练。然而,由于每次更新需更多样本且执行时间更长,其是否真能减少达到目标所需的实际耗时仍不明确。本文研究大语言模型强化学习中的这一权衡,通过分离算法与系统层面的影响进行分析。在算法层面,固定累计样本数并重调依赖批次的超参数,发现批量大小在有限范围内可产生近似不变的学习曲线。在系统层面,利用推理生成与训练之间的计算不对称性:自回归生成在低并发时受内存带宽限制,而训练耗时与处理的词元数近似成正比。结合两者,得出直接决策规则:只有当吞吐增益超过样本消耗惩罚时,大批次才可缩短时间至目标。实验基于GRPO和PPO验证了该分解的两方面。算法上,使用Adam时按平方根缩放学习率,可在一定批量范围内实现近似批量无关的学习曲线;系统上,大批次使生成吞吐最高提升2.29倍。在GRPO中,结合更高吞吐与学习率重调,时间至目标最多减少29%;而仅增大批次不重调学习率,则因样本成本过高反而更慢。
原文摘要 · Abstract (English)
Larger batches reduce the variance of stochastic gradients per update and are therefore often expected to accelerate training. Yet whether this statistical benefit translates into lower wall-clock time-to-target remains unclear, because each update consumes more samples and may take longer to execute. We study this tradeoff in reinforcement learning for large language models. We separate its algorithmic and systems effects by comparing learning and execution along their natural axes. At the algorithmic level, we compare configurations at equal cumulative sample counts while retuning batch-dependent hyperparameters. Over a bounded range of batch sizes, this procedure yields an approximately batch-size-invariant family whose members follow similar sample-indexed learning trajectories. At the systems level, we exploit the computational asymmetry between rollout generation and training: autoregressive generation is often memory-bandwidth-bound at low concurrency, whereas training work scales approximately with the number of processed tokens. Combining these two views yields a direct decision rule: a larger-batch configuration reduces time-to-target only when its throughput gain exceeds its samples-to-target penalty. Experiments with GRPO and PPO support both sides of this decomposition. At the algorithmic level, square-root learning-rate scaling with Adam produces approximately batch-size-invariant learning curves over a bounded range of batch sizes. At the systems level, larger batches improve generation throughput by up to 2.29x on fixed hardware. In GRPO, combining higher throughput with learning-rate retuning reduces time-to-target by up to 29%, whereas increasing the batch without retuning is slower despite its higher throughput.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。