通过在线可塑性提升神经进化算法的样本效率与鲁棒性
Provably Sub-Linear Two-Timescale NeuroEvolution with Online Plasticity

- 将架构搜索与在线权重调整分时进行,实现双时间尺度优化
- 在四个标准控制任务上,比纯NEAT提升最终适应度并降低方差
- 首次提供理论证明:该框架可达到次线性遗憾,适合强化学习初学者
神经进化增广拓扑(NEAT)是一种广泛用于学习神经网络架构与权重的进化算法,适用于控制任务。然而,传统离线优化直接搜索连接强度,在高维权重空间和复杂连续控制问题中效率较低。结合神经进化与在线学习的混合方法可缓解此问题,但其理论性质尚不明确。本文首次对通用神经进化在线学习(NEOL)框架进行后悔率分析,该框架将学习分为外层架构搜索与内层基于奖励调制的在线权重适应两个时间尺度。在弱条件下,证明了NEOL可实现次线性后悔。实验表明,在固定交互预算下,基于NEAT的NEOL实现比纯NEAT获得更高最终适应度且方差更低,并在多个任务上媲美强基线强化学习方法。结果经威尔科克森秩和检验与消融研究验证。整体表明,在线可塑性可显著提升双时间尺度神经进化算法的样本效率与鲁棒性。代码已公开于https://github.com/boobaa2001/NeuroEvolutionOnlineLearningNEOL。
原文摘要 · Abstract (English)
NeuroEvolution of Augmenting Topologies (NEAT) is a widely used neuroevolution algorithm for learning neural network architectures and weights for control tasks. However, standard offline optimisation searches for connection strengths directly, which can scale poorly in high-dimensional weight spaces and more difficult continuous control problems. Hybrid methods that combine neuroevolution with online learning can address this challenge, but their theoretical properties remain underexplored. This paper gives the first regret analysis for a general NeuroEvolutionary Online Learning (NEOL) framework, which decouples learning into two timescales: an outer loop for architecture search and an inner loop for online weight adaptation via rewardmodulated plasticity. Under mild conditions, we prove that NEOL achieves sublinear regret. Empirically, under fixed interaction budgets on four standard control benchmarks, a NEAT-based NEOL implementation achieves higher final fitness and lower variance than pure NEAT, and is competitive with strong reinforcement learning (RL) baselines on several tasks. The results are supported byWilcoxon rank-sum tests and ablation studies. Overall, the findings show that online plasticity can improve the sample efficiency and robustness of two-timescale neuroevolution. Code is available at https://github.com/boobaa2001/NeuroEvolution Online Learning NEOL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。