用进化算法自动发现强化学习中的动态奖励策略,提升稀疏奖励任务表现。
Evolutionary Discovery of Developmental Reward Schedules in Deep Reinforcement Learning

- 通过进化算法优化三类生物启发动机的时变权重组合。
- 在DoorKey-6x6任务上,最优方法比外部奖励基线提升11.4%平均性能。
- 发现新颖性早期主导奖励信号,与预设发育顺序相反。
强化学习中奖励结构通常人工设计且固定不变,限制了动机优先级的演变探索。本文提出一种进化框架,通过时间可变权重组合三种生物启发动机成分——自主性、新颖性和反应性,动态调整训练过程中的激励方向。在两个稀疏奖励的MiniGrid任务(DoorKey-6x6和KeyCorridorS3R1)上评估了四种进化算法(CMA-ES、xNES、DE、L-SHADE)与外部激励基线及三种手工设计方法的性能。在DoorKey-6x6任务上,所有进化方法均优于非进化基线,其中L-SHADE表现最佳,相对平均性能提升约11.4%;在KeyCorridorS3R1任务上,CMA-ES表现最优,其余进化方法泛化能力较弱且不稳定。有趣的是,发现的奖励调度在两个任务中均显示新颖性作为早期主导信号,与预设的发育顺序不一致。结果表明,进化优化是深度强化学习中发现发育型奖励调度的有力方法,且计算环境中进化的最优解可能不同于生物学中的最优解。
原文摘要 · Abstract (English)
The temporal structure of reward composition in reinforcement learning (RL) is typically hand-designed and held fixed throughout training, leaving the progression of motivational priorities largely unexplored. In this work, we propose an evolutionary framework for discovering developmental reward schedules, in which three distinct biologically inspired motivational components -- agency, novelty, and reactivity -- are combined through time-varying weights that dynamically shift over the course of training. Evaluated on two sparse-reward MiniGrid tasks: DoorKey-6x6 and KeyCorridorS3R1, our framework compares the generalizability of four evolutionary algorithms: CMA-ES, xNES, DE, and L-SHADE against an extrinsically motivated baseline (our main comparison point), and three additional hand-designed methods. On DoorKey-6x6, all evolved methods outperform the non-evolved baselines, with L-SHADE achieving the best performance -- an approximate relative mean improvement of 11.4% over the extrinsic only baseline. On KeyCorridorS3R1, CMA-ES achieves the best overall performance, with the remaining evolved methods showing weaker and less reliable generalization capability compared to the extrinsic only baseline. Interestingly, the discovered schedules diverge from our defined developmental ordering, with novelty consistently emerging as the dominant early signal during training, across both tasks. Collectively, our results position evolutionary optimization as a promising approach for developmental reward schedule discovery in deep reinforcement learning, and suggest that what evolution finds to be optimal in computational settings may differ from what it finds to be optimal in biology. The code for this project can be found at: https://github.com/alannadels/Evolutionary_RL.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。