用插值近似代理策略，让异步强化学习训练快1.8倍。
A-3PO: Accelerating Asynchronous LLM Training with Staleness-aware Proximal Policy Approximation
- 用线性插值替代显式计算代理策略，降低计算开销。
- 在相同性能下实现1.8倍训练加速，保持学习稳定性。
- 适合大规模语言模型的异步强化学习训练场景。
解耦PPO是应对异步强化学习中高数据过时问题的成功算法。其使用的解耦损失通过引入代理策略，将离策略修正（重要性权重）与策略更新约束（信任区域）分离，提升了学习稳定性。然而，代理策略需在每个训练步骤额外进行一次前向传播，对大语言模型训练造成计算负担。我们发现，由于代理策略仅作为行为策略与目标策略间的信任区域锚点，可通过简单插值近似实现，无需显式计算。我们提出A-3PO（APproximated Proximal Policy Optimization），完全消除该开销，在保持相当性能的前提下实现1.8倍训练加速。代码与即用示例已开源至AReaL系统：https://github.com/areal-project/AReaL/blob/v1.0.0.rc1/docs/algorithms/prox_approx.md
原文摘要 · Abstract (English)
Decoupled PPO has been a successful reinforcement learning (RL) algorithm to deal with the high data staleness under the asynchronous RL setting. Decoupled loss used in decoupled PPO improves coupled-loss style of algorithms' (e.g., standard PPO, GRPO) learning stability by introducing a proximal policy to decouple the off-policy correction (importance weight) from the policy update constraint (trust region). However, the proximal policy requires an extra forward pass through the model at each training step, creating a computational overhead for large language models training. We observe that since the proximal policy only serves as a trust region anchor between the behavior and target policies, we can approximate it through simple interpolation without explicit computation. We call this approach A-3PO (APproximated Proximal Policy Optimization). A-3PO eliminates this overhead, accelerating training by 1.8x speedup while maintaining comparable performance. Code \& off-the-shelf example are contributed to the open-source RL training system AReaL at: https://github.com/areal-project/AReaL/blob/v1.0.0.rc1/docs/algorithms/prox_approx.md
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。