通过错开环境重置时间，提升大规模并行强化学习的稳定性和效率
Staggered Environment Resets Improve Massively Parallel On-Policy Reinforcement Learning
- 采用错时重置策略，避免同步重置带来的非平稳性问题
- 在高维机器人任务中实现更快收敛与更高样本效率
- 特别适合大规模并行训练场景，扩展性优于传统同步重置
大规模并行GPU仿真环境通过加速数据收集推动了强化学习研究，尤其适用于PPO等在线策略算法。为最大化吞吐量，通常采用短轨迹进行每轮策略更新，从而提高更新-数据比（UTD）。然而我们发现，在此设置下，标准同步重置会引入有害的非平稳性，扭曲学习信号并导致训练不稳定。本文提出错时重置：让环境在任务周期内不同时间点被初始化和重置，从而生成具有更丰富时间多样性的训练批次，降低同步轨迹带来的非平稳性。我们在简化玩具环境中验证了该方法的有效性，并将其应用于高维机器人任务，显著提升了样本效率、加快了壁挂时间收敛速度，且最终性能更强。此外，该方法在更多并行环境下的扩展性优于朴素同步重置。
原文摘要 · Abstract (English)
Massively parallel GPU simulation environments have accelerated reinforcement learning (RL) research by enabling fast data collection for on-policy RL algorithms like Proximal Policy Optimization (PPO). To maximize throughput, it is common to use short rollouts per policy update, increasing the update-to-data (UTD) ra- tio. However, we find that, in this setting, standard synchronous resets introduce harmful nonstationarity, skewing the learning signal and destabilizing training. We introduce staggered resets, a simple yet effective technique where environments are initialized and reset at varied points within the task horizon. This yields training batches with greater temporal diversity, reducing the nonstationarity induced by synchronized rollouts. We characterize dimensions along which RL environments can benefit significantly from staggered resets through illustrative toy environ- ments. We then apply this technique to challenging high-dimensional robotics environments, achieving significantly higher sample efficiency, faster wall-clock convergence, and stronger final performance. Finally, this technique scales better with more parallel environments compared to naive synchronized rollouts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。