融合进化算法与策略梯度，提升强化学习的样本效率与可扩展性。
Evolutionary Policy Optimization
- 用潜变量控制的种群共享网络参数，增强多样性与记忆效率。
- 在复杂任务中实现更高样本效率与更优最终性能，支持大规模并行。
- 适合需要高效探索与稳定训练的机器人控制场景。
在线强化学习算法因优异的渐近性能和训练稳定性而广泛应用，但难以扩展至大批次，因额外并行环境产生冗余数据，受限于策略带来的多样性不足。相反，进化算法（EAs）天然具备良好可扩展性，通过基于种群的随机搜索促进探索，但常存在样本效率低的问题。本文提出进化策略优化（EPO），融合进化算法的可扩展性与多样性优势，以及策略梯度的高性能与稳定性。EPO维护一个由潜变量条件化的代理种群，共享演员-评论家网络参数以保证一致性并节省内存，并将多样化的经验聚合为一个主代理。在灵巧操作、足式运动和经典控制等任务中，EPO在样本效率、渐近性能和可扩展性方面均优于当前最优基线。
原文摘要 · Abstract (English)
On-policy reinforcement learning (RL) algorithms are widely used for their strong asymptotic performance and training stability, but they struggle to scale with larger batch sizes, as additional parallel environments yield redundant data due to limited policy-induced diversity. In contrast, Evolutionary Algorithms (EAs) scale naturally and encourage exploration via randomized population-based search, but are often sample-inefficient. We propose Evolutionary Policy Optimization (EPO), a hybrid algorithm that combines the scalability and diversity of EAs with the performance and stability of policy gradients. EPO maintains a population of agents conditioned on latent variables, shares actor-critic network parameters for coherence and memory efficiency, and aggregates diverse experiences into a master agent. Across tasks in dexterous manipulation, legged locomotion, and classic control, EPO outperforms state-of-the-art baselines in sample efficiency, asymptotic performance, and scalability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。