调大并行环境比拉长采样步数更能提升强化学习性能
The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networks
- 用更多并行环境收集数据，比延长单条轨迹更有效
- 数据量越大，最终性能越高，且在多种设置下稳定提升
- 对网络结构和超参数敏感度低，适合大规模部署
并行智能体用于数据收集是强化学习中的有效方法。数据收集方式（并行环境数量与轨迹长度）引发偏差-方差权衡；而对采集数据的训练轮次需在样本效率与过拟合间取得平衡。本文针对最流行的PPO算法进行实证分析，揭示其与网络可塑性及优化稳定性之间的关联。研究考察了不同网络架构下的表现，以及扩展数据时的超参数敏感性。结果表明，更大规模的数据集能在多种场景中提升最终性能，且增加并行环境数的效果优于延长回滚长度。这些发现强调了数据收集策略对智能体性能的关键作用。
原文摘要 · Abstract (English)
The use of parallel actors for data collection has been an effective technique used in reinforcement learning (RL) algorithms. The manner in which data is collected in these algorithms, controlled via the number of parallel environments and the rollout length, induces a form of bias-variance trade-off; the number of training passes over the collected data, on the other hand, must strike a balance between sample efficiency and overfitting. We conduct an empirical analysis of these trade-offs on PPO, one of the most popular RL algorithms that uses parallel actors, and establish connections to network plasticity and, more generally, optimization stability. We examine its impact on network architectures, as well as the hyper-parameter sensitivity when scaling data. Our analyses indicate that larger dataset sizes can increase final performance across a variety of settings, and that scaling parallel environments is more effective than increasing rollout lengths. These findings highlight the critical role of data collection strategies in improving agent performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。