针对异步强化学习中延迟导致的不稳定问题,提出自适应信任区域方法。
Stale but Stable: Staleness-Adaptive Trust Regions for Stabilizing Asynchronous Reinforcement Learning

- 基于采样对数似然比构建延迟代理,动态识别高延迟样本
- 在延迟高的批次中仅收缩更新方向,提升稳定性,滞后8时仍达34.79
- 适合大规模异步训练场景,尤其适用于高延迟环境下的策略优化
异步强化学习通过解耦生成与优化提升吞吐量,但由此带来的延迟不可避免,且受策略滞后、引擎延迟和专家路由混合影响。从信任域视角看,这种不匹配至关重要:有限时域改进界中,训练-推理偏差主导近似误差,而PPO裁剪仅限制采样外推更新,仅作为采样替代而非完整策略约束。因此,在最需要控制延迟更新的异步环境中,高延迟更新仍可能被弱控制。本文提出滞留自适应信任区域(SAT),利用脱离的采样对数似然比作为实用的延迟代理,通过基于延迟的核函数缩放识别每批中的高失配尾部,并仅使用有效收缩因子收缩名义PPO区间的符号选择端点。该设计在普通样本上保持基线行为,同时在新截获的外推带中使更新更保守。我们在基于Qwen3-30B-A3B-Base的全解耦异步强化学习设置下评估SAT,采用SGLang作为推理引擎,Megatron作为训练流水线。结果表明,SAT-GSPO w/ R3在滞后1时达到35.83,滞后8时达34.79,为当前最优;整体显示,将裁剪区间与观测到的延迟异质性对齐是稳定异步环境的有效方式。
原文摘要 · Abstract (English)
Asynchronous reinforcement learning improves throughput by decoupling rollout generation from optimization, but the resulting staleness is an inevitable byproduct, compounded jointly by policy lag, engine delays, and mixture-of-experts routing. From a trust-region perspective, this mismatch is critical: in the finite-horizon improvement bound, training-inference divergence governs the approximation error, whereas PPO clipping only gates sampled outward updates and therefore acts as a sampled surrogate rather than a full-policy constraint. As a result, the high-staleness update can remain weakly controlled in exactly the asynchronous regime where stale rollouts matter most. We introduce the Staleness-Adaptive Trust Region (SAT), which uses the detached sampled log-ratio as a practical staleness proxy, identifies the high-mismatch tail within each batch through Staleness-based kernel function scaling, and contracts only the sign-selected endpoint of the nominal PPO interval using Effective contraction factors. This design preserves the baseline behavior on ordinary tokens, while making the update more conservative exactly on newly intercepted outward bands. We evaluate SAT in a fully decoupled asynchronous reinforcement learning setup built on Qwen3-30B-A3B-Base, leveraging SGLang as the inference engine and Megatron as the training pipeline. In this setting, SAT-GSPO w/ R3 attains the best observed AIME24 avg@8, reaching 35.83 at lag 1 and 34.79 at lag 8, while SAT-GSPO reaches 34.17 at lag 1. More broadly, the results indicate that aligning the clip interval with observed staleness heterogeneity is an effective way to stabilize the reported asynchronous regime.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。