用大模型自动演化出新强化学习算法，性能媲美SAC等经典方法。
Evolutionary Discovery of Reinforcement Learning Algorithms via Large Language Models
- 用大模型生成可执行的学习更新规则，直接搜索完整训练流程。
- 在Gymnasium上训练后，新算法性能接近SAC、PPO等基准方法。
- 排除经典结构，发现非标准学习规则，适合算法创新研究者。
强化学习算法由其学习更新规则定义，通常手工设计且固定不变。本文提出一种演化框架，直接在可执行的更新规则空间中搜索完整的训练过程。该方法基于REvolve系统，利用大语言模型作为生成变异算子，并将其从奖励函数发现扩展至算法发现。为促进非标准学习规则的涌现，搜索过程排除了典型机制，如演员-评论家结构、时序差分损失和价值自举。由于强化学习算法对内部标量参数高度敏感，我们引入后演化优化阶段，由大语言模型为每个演化出的更新规则提出可行的超参数范围。在多个Gymnasium基准上通过完整训练评估，所发现的算法在性能上与SAC、PPO、DQN和A2C等成熟基线相当。
原文摘要 · Abstract (English)
Reinforcement learning algorithms are defined by their learning update rules, which are typically hand-designed and fixed. We present an evolutionary framework for discovering reinforcement learning algorithms by searching directly over executable update rules that implement complete training procedures. The approach builds on REvolve, an evolutionary system that uses large language models as generative variation operators, and extends it from reward-function discovery to algorithm discovery. To promote the emergence of nonstandard learning rules, the search excludes canonical mechanisms such as actor--critic structures, temporal-difference losses, and value bootstrapping. Because reinforcement learning algorithms are highly sensitive to internal scalar parameters, we introduce a post-evolution refinement stage in which a large language model proposes feasible hyperparameter ranges for each evolved update rule. Evaluated end-to-end by full training runs on multiple Gymnasium benchmarks, the discovered algorithms achieve competitive performance relative to established baselines, including SAC, PPO, DQN, and A2C.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。