解决大模型强化学习异步训练中的方差爆炸问题，提升稳定性和效率。
Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMs
- 基于有效样本量动态调整学习率，抑制过时数据带来的噪声更新。
- 无需批评者模型，通过闭式最小方差基线降低估计方差，支持128步异步训练。
- 在长序列工具使用任务中实现同步性能，训练速度提升2.5倍，适合大规模部署。
异步强化学习在大模型后训练中日益重要，通过解耦生成与策略更新显著提升吞吐量。然而，主流的策略梯度方法（如REINFORCE、GRPO）在高异步场景下表现不佳：过时的轨迹导致重尾重要性权重，少数轨迹主导更新，使策略梯度估计方差剧增。我们在数学、推理和工具使用基准上系统分析发现，方差增长可由有效样本量（ESS）下降可靠预测，而现有稳定化方法未能有效应对。为此，我们提出方差控制策略优化（VCPO），其一，动态根据ESS缩放学习率以抑制不可靠更新；其二，在无需批评者模型的前提下，采用闭式最小方差基线处理离策略情形，开销极低。实验证明，无论在数学还是通用推理任务中，VCPO均能实现鲁棒稳定的异步训练，即使在128步离策略条件下依然有效。在长周期工具使用任务中，其性能媲美同步训练，同时将训练时间缩短2.5倍。代码已开源：https://github.com/mit-han-lab/vcpo。
原文摘要 · Abstract (English)
Asynchronous reinforcement learning has become increasingly central to scaling LLM post-training, delivering major throughput gains by decoupling rollout generation from policy updates. However, widely used policy-gradient objectives such as REINFORCE and GRPO suffer under high asynchrony: stale rollouts produce heavy-tailed importance weights, so a small number of trajectories dominate updates and the policy-gradient estimator becomes markedly higher variance. Through systematic analysis on math, reasoning, and tool-use benchmarks, we find that this increasing variance is reliably predicted by collapsing effective sample size (ESS), which prior stabilization methods largely fail to address. Motivated by this diagnosis, we introduce $\textbf{V}$ariance $\textbf{C}$ontrolled $\textbf{P}$olicy $\textbf{O}$ptimization ($\textbf{VCPO}$), a method that (i) dynamically scales the learning rate with ESS to dampen unreliable updates and (ii) applies a closed-form minimum-variance baseline for off-policy settings, without a critic model and adding minimal overhead. Empirically, across math and general reasoning benchmarks, this enables robustly stable asynchronous training compared to previous stabilization and algorithmic methods, even in highly off-policy regimes (128 steps off-policy). In a long-horizon, tool-use task, VCPO matches synchronous performance while delivering a 2.5$\times$ speedup in training time. Code is available at: https://github.com/mit-han-lab/vcpo
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。