分布式强化学习新框架，降低同步开销，提升训练效率。
Adaptive Policy Synchronization for Scalable Reinforcement Learning
- 将环境操作移至远程工作节点，学习保持集中，实现轻量级分布式运行。
- 自适应策略同步机制减少通信频率，性能与传统方法相当但开销更低。
- 兼容主流算法，改造少，适合大规模强化学习场景的开发者使用。
大规模强化学习通常需要在多台机器上运行环境，但多数框架将模拟、训练和基础设施绑定为刚性系统。本文提出ClusterEnv，一种轻量级分布式环境执行接口，保留熟悉的Gymnasium API。ClusterEnv采用DETACH模式，将环境reset()和step()操作移至远程工作节点，同时保持学习过程集中。为减少策略过时且避免频繁通信，提出自适应策略同步（APS），仅当工作节点与中心学习器的策略差异超过阈值时才请求更新。ClusterEnv支持在线与离线策略方法，可无缝集成到现有训练代码中，集群部署高效。在离散控制任务上的实验表明，APS在保持性能的同时显著降低同步开销。源码已开源：https://github.com/rodlaf/ClusterEnv。
原文摘要 · Abstract (English)
Scaling reinforcement learning (RL) often requires running environments across many machines, but most frameworks tie simulation, training, and infrastructure into rigid systems. We introduce ClusterEnv, a lightweight interface for distributed environment execution that preserves the familiar Gymnasium API. ClusterEnv uses the DETACH pattern, which moves environment reset() and step() operations to remote workers while keeping learning centralized. To reduce policy staleness without heavy communication, we propose Adaptive Policy Synchronization (APS), where workers request updates only when divergence from the central learner grows too large. ClusterEnv supports both on- and off-policy methods, integrates into existing training code with minimal changes, and runs efficiently on clusters. Experiments on discrete control tasks show that APS maintains performance while cutting synchronization overhead. Source code is available at https://github.com/rodlaf/ClusterEnv.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。