将强化学习与遗传算法结合,提升组合优化的求解质量与训练效率。
Synergizing Reinforcement Learning and Genetic Algorithms for Neural Combinatorial Optimization
- 用遗传操作演化强化学习生成的解,再回流到训练中增强探索。
- 在旅行商等经典问题上,解的质量和收敛速度均显著优于基线方法。
- 框架通用性强,适用于多种主流深度强化学习求解器,适合优化任务研究者。
组合优化问题因离散结构和指数级解空间而极具挑战。近年来,深度强化学习(DRL)可直接从数据中学习启发式策略,但普遍存在探索有限、易陷入局部最优的问题。而遗传算法(GAs)虽具备强大全局搜索能力,却通常样本效率低且计算成本高。本文提出一种通用且即插即用的进化增强机制(EAM),将DRL的学习效率与GAs的全局搜索能力相结合。EAM通过学习策略生成解,并利用领域特定的交叉、变异等遗传操作进行优化,再将优质解选择性地回注入策略训练流程,从而提升探索能力并加速收敛。我们进一步提供了理论分析,建立了演化解分布与策略分布间KL散度的上界,确保策略更新稳定有效。EAM对模型无偏见,可无缝集成于注意力模型、POMO、SymNCO等先进DRL求解器。在旅行商问题(TSP)、容量约束旅行商问题(CVRP)、部分覆盖旅行商问题(PCTSP)及开放路径问题(OP)等基准测试中,结果表明EAM显著提升了方案质量与训练效率,优于多个竞争基线。
原文摘要 · Abstract (English)
Combinatorial optimization problems are notoriously challenging due to their discrete structure and exponentially large solution space. Recent advances in deep reinforcement learning (DRL) have enabled the learning heuristics directly from data. However, DRL methods often suffer from limited exploration and susceptibility to local optima. On the other hand, evolutionary algorithms such as Genetic Algorithms (GAs) exhibit strong global exploration capabilities but are typically sample inefficient and computationally intensive. In this work, we propose the Evolutionary Augmentation Mechanism (EAM), a general and plug-and-play framework that synergizes the learning efficiency of DRL with the global search power of GAs. EAM operates by generating solutions from a learned policy and refining them through domain-specific genetic operations such as crossover and mutation. These evolved solutions are then selectively reinjected into the policy training loop, thereby enhancing exploration and accelerating convergence. We further provide a theoretical analysis that establishes an upper bound on the KL divergence between the evolved solution distribution and the policy distribution, ensuring stable and effective policy updates. EAM is model-agnostic and can be seamlessly integrated with state-of-the-art DRL solvers such as the Attention Model, POMO, and SymNCO. Extensive results on benchmark problems (e.g., TSP, CVRP, PCTSP, and OP) demonstrate that EAM significantly improves both solution quality and training efficiency over competitive baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。