用进化算法自动发现比Adam更优的深度学习优化器。
Evolving Deep Learning Optimizers
- 将优化器编码为基因，通过演化搜索组合更新项和超参数。
- 在多个视觉任务上表现优于Adam，CIFAR-10提升7.7%。
- 发现新设计规律：低动量、禁用偏差校正、启用学习率预热与余弦衰减。
我们提出一种基于遗传算法的框架，用于自动发现深度学习优化算法。该方法将优化器编码为基因，包含梯度、动量、RMS归一化、Adam式自适应项及基于符号的更新等基本更新项，以及超参数和调度选项。在50代、每代50个个体的演化搜索中，于多个视觉任务上评估后，发现一种新型优化器，在综合性能上比Adam高出2.6%，在CIFAR-10上实现7.7%的相对提升。该优化器结合了基于符号的梯度项与自适应动量估计，使用低于Adam的动量系数（β₁=0.86，β₂=0.94），并显著禁用偏差校正，同时支持学习率预热和余弦衰减。结果表明，演化搜索可发现具有竞争力的优化算法，并揭示出与人工设计不同的优化器设计规律。代码已开源：https://github.com/mmarfinetz/evo-optimizer。
原文摘要 · Abstract (English)
We present a genetic algorithm framework for automatically discovering deep learning optimization algorithms. Our approach encodes optimizers as genomes that specify combinations of primitive update terms (gradient, momentum, RMS normalization, Adam-style adaptive terms, and sign-based updates) along with hyperparameters and scheduling options. Through evolutionary search over 50 generations with a population of 50 individuals, evaluated across multiple vision tasks, we discover an evolved optimizer that outperforms Adam by 2.6% in aggregate fitness and achieves a 7.7% relative improvement on CIFAR-10. The evolved optimizer combines sign-based gradient terms with adaptive moment estimation, uses lower momentum coefficients than Adam ($β_1$=0.86, $β_2$=0.94), and notably disables bias correction while enabling learning rate warmup and cosine decay. Our results demonstrate that evolutionary search can discover competitive optimization algorithms and reveal design principles that differ from hand-crafted optimizers. Code is available at https://github.com/mmarfinetz/evo-optimizer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。