TianJi通过松散任务依赖实现强化学习高效并行,训练速度提升超7倍。
Highly Parallelized Reinforcement Learning Training with Relaxed Assignment Dependencies
- 松耦合任务设计+事件驱动通信,打破传统依赖瓶颈
- 在8节点下比XingTian快1.6倍,吞吐提升7.13倍
- 适用于在线策略算法,适合大规模强化学习部署
随着对高性能智能体需求的增长,深度强化学习(DRL)训练复杂度持续上升。当前系统受限于子任务间的数据分配依赖,难以充分并行化。为此,我们提出高吞吐分布式强化学习训练系统TianJi,通过放松子任务组件间的分配依赖,实现事件驱动的异步通信,并保持组件边界清晰。为应对松散依赖带来的收敛不确定性,TianJi设计了一种基于样本生产与消费平衡的分布式策略,控制样本延迟以修正质量,保障收敛性。实验表明,TianJi相比同类系统最高可实现4.37倍的收敛加速;在8个计算节点下,收敛时间提速1.6倍,吞吐提速7.13倍;数据传输效率接近硬件极限。该系统在在线策略算法中也表现优异,相比RLlib和XingTian分别获得4.36和2.95倍的收敛加速。项目开源:https://github.com/HiPRL/TianJi.git。
原文摘要 · Abstract (English)
As the demands for superior agents grow, the training complexity of Deep Reinforcement Learning (DRL) becomes higher. Thus, accelerating training of DRL has become a major research focus. Dividing the DRL training process into subtasks and using parallel computation can effectively reduce training costs. However, current DRL training systems lack sufficient parallelization due to data assignment between subtask components. This assignment issue has been ignored, but addressing it can further boost training efficiency. Therefore, we propose a high-throughput distributed RL training system called TianJi. It relaxes assignment dependencies between subtask components and enables event-driven asynchronous communication. Meanwhile, TianJi maintains clear boundaries between subtask components. To address convergence uncertainty from relaxed assignment dependencies, TianJi proposes a distributed strategy based on the balance of sample production and consumption. The strategy controls the staleness of samples to correct their quality, ensuring convergence. We conducted extensive experiments. TianJi achieves a convergence time acceleration ratio of up to 4.37 compared to related comparison systems. When scaled to eight computational nodes, TianJi shows a convergence time speedup of 1.6 and a throughput speedup of 7.13 relative to XingTian, demonstrating its capability to accelerate training and scalability. In data transmission efficiency experiments, TianJi significantly outperforms other systems, approaching hardware limits. TianJi also shows effectiveness in on-policy algorithms, achieving convergence time acceleration ratios of 4.36 and 2.95 compared to RLlib and XingTian. TianJi is accessible at https://github.com/HiPRL/TianJi.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。