将进化计算与梯度优化结合，提升强化学习的探索效率与性能。
Evolutionary Policy Optimization
- 融合进化算法的全局探索与策略梯度的局部优化能力。
- 在Atari Pong和Breakout上比传统方法提升策略质量与采样效率。
- 适合需要兼顾探索与精细优化的强化学习任务。
强化学习中的关键挑战是如何在不牺牲样本效率的前提下平衡探索与利用。策略梯度（PG）方法通过基于梯度的精细化优化擅长利用，但常因聚焦局部搜索而难以探索。相反，进化计算（EC）方法擅长全局探索，却缺乏利用机制。为此，本文提出进化策略优化（EPO），一种将神经进化与策略梯度结合的混合算法。EPO同时利用EC的探索能力和PG的利用优势，有效解决强化学习中的探索-利用困境。在Atari Pong和Breakout基准测试中，实验结果表明，相较于标准的PG和EC方法，EPO在策略质量和样本效率上均有提升，适用于既需探索又需局部优化的任务。
原文摘要 · Abstract (English)
A key challenge in reinforcement learning (RL) is managing the exploration-exploitation trade-off without sacrificing sample efficiency. Policy gradient (PG) methods excel in exploitation through fine-grained, gradient-based optimization but often struggle with exploration due to their focus on local search. In contrast, evolutionary computation (EC) methods excel in global exploration, but lack mechanisms for exploitation. To address these limitations, this paper proposes Evolutionary Policy Optimization (EPO), a hybrid algorithm that integrates neuroevolution with policy gradient methods for policy optimization. EPO leverages the exploration capabilities of EC and the exploitation strengths of PG, offering an efficient solution to the exploration-exploitation dilemma in RL. EPO is evaluated on the Atari Pong and Breakout benchmarks. Experimental results show that EPO improves both policy quality and sample efficiency compared to standard PG and EC methods, making it effective for tasks that require both exploration and local optimization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。