用智能体自动进化出更优强化学习网络结构。
EVOM: Agentic Meta-Evolution of Actor-Critic Architectures for Reinforcement Learning
- 设计智能体通过元进化迭代优化网络架构程序。
- 在Ant-v4和HalfCheetah-v4上性能超越现有方法。
- 纯设计智能体不参与策略执行，保证高效性。
在强化学习的演员-评论家框架中，网络结构通常手动设计。自动化设计面临两大挑战：每个候选架构需训练后才能评估，且设计空间开放无边界。为此，我们提出EVOM，一种基于智能体的元进化框架，用于发现高性能的演员-评论家架构。将架构搜索建模为双层优化问题：内层通过低精度近端策略优化（PPO）训练权重，外层通过迭代优化架构程序实现元进化。关键在于，外层由基于大语言模型（LLM）的设计智能体驱动，该智能体仅负责架构设计，与策略执行和环境控制完全解耦。实验表明，EVOM在Ant-v4和HalfCheetah-v4上优于人工设计基线、LLM引导的随机搜索及当前最优的LLM引导程序化策略搜索方法MLES。消融研究验证了元进化循环和LLM设计智能体对最终性能不可或缺。
原文摘要 · Abstract (English)
In actor-critic reinforcement learning, network architectures are typically manually designed. Automating this design is challenging because each candidate must be trained before evaluation, and the design space is open-ended. To address these challenges, we introduce EVOM, an agentic meta-evolution framework for discovering high-performance actor-critic architectures. We frame architecture search as a bi-level optimization: an inner loop trains weights via the low-fidelity proximal policy optimization (PPO), while an outer loop drives meta-evolution by iteratively refining architecture programs. Crucially, this outer loop is powered by an LLM-based design agent that operates purely as an architecture designer, completely decoupled from policy execution and environment control. Experiments reveal that EVOM outperforms the manually designed baseline, an LLM-guided random search, and the state-of-the-art LLM-guided programmatic policy search method MLES, delivering superior performance on Ant-v4 and HalfCheetah-v4. Ablation studies validate that both the meta-evolution loop and the LLM Design Agent are indispensable for final performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。