用深度强化学习找到可解释的多参数控制策略,提升进化算法性能。
Discovering Interpretable Multi-Parameter Control Policies for Evolutionary Algorithms Using Deep Reinforcement Learning

- 设计新方法解决多参数设置下的收敛难题
- 提出符号化控制策略,性能优于现有基线
- 适合对算法可解释性与优化效率有要求的研究者
尽管深度强化学习在进化算法参数控制中应用日益广泛,但严谨的理论分析仍主要局限于单参数场景,原因在于难以推导出有效且可解释的多参数策略以支持形式化研究。本文以(1+($λ$,$λ$))-遗传算法优化OneMax问题为典型案例,该问题已被证明动态控制可实现超常数级加速。我们首先表明标准方法在此多参数设置下难以收敛,进而提出无需依赖具体算法的改进:动作空间分解、奖励偏移和长时折扣。在这些改进基础上,对比多种深度强化学习方法发现,Double Deep Q-Network(DDQN)能避免近端策略优化(PPO)中的策略坍塌现象,生成适合后续分析的训练轨迹。关键在于,我们突破神经网络的“黑箱”局限,将学习到的行为提炼为透明的符号化控制策略。该策略不仅便于未来理论分析,且在多种问题规模下持续超越现有基准,表现优异。
原文摘要 · Abstract (English)
While deep Reinforcement Learning (deep-RL) has been increasingly applied to parameter control in evolutionary algorithms, rigorous theoretical analysis of parameter control remains largely restricted to single-parameter settings, owing to the difficulty of deriving effective, interpretable multi-parameter policies amenable to formal study. We demonstrate how deep-RL can be leveraged to overcome this barrier, using the (1+($λ$,$λ$))-genetic algorithm optimizing OneMax, one of the few problems where a super-constant speedup of dynamic control has been formally proven, as a representative case study. We first show that standard approaches struggle to converge in this multi-parameter setting, and introduce algorithm-agnostic enhancements targeting action-space decomposition, reward shifting, and long-horizon discounting. With these in place, we compare common deep-RL methods and find that Double Deep Q-Networks uniquely avoid the policy collapse observed in Proximal Policy Optimization, yielding trajectories suitable for downstream analysis. Crucially, we move beyond the ``black-box'' nature of neural networks by distilling the learned behaviors into a transparent, symbolic control policy. This resulting policy does not only offer interpretability for future theoretical analysis but also yields exceptional performance, consistently outperforming existing baselines across a wide range of problem sizes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。