提出可大规模并行的快速质量多样性算法,生成高质量多样化神经网络策略。
Scaling Policy Gradient Quality-Diversity with Massive Parallelization via Behavioral Variations
- 基于时序性能指标进行行为变异,用策略梯度映射到解空间
- 单卡250秒内完成高维策略生成,比现有方法快5倍
- 无需集中式训练,适合大规模并行部署
质量-多样性优化是一类旨在生成多样且高性能解集的进化算法。MAP-Elites(ME)是其中典型代表,广泛应用于进化机器人等领域。然而,传统ME依赖遗传算法的随机突变,难以高效演化高维解。已有方法尝试引入基于梯度的算子如策略梯度或自然进化策略,虽在神经演化中表现良好,但普遍存在训练速度慢、难以实现大规模并行等问题,受限于高计算开销或集中式演员-评论家训练。本文提出一种新型快速、样本高效的ME算法——ASCII-ME,其不依赖集中式演员-评论家训练,通过基于时间步性能指标的行为变异,并利用策略梯度将变异映射为解决方案。实验表明,ASCII-ME可在单个GPU上于250秒内生成多样化且高性能的深度神经网络策略,平均运行速度较最先进方法快五倍,同时保持竞争力的样本效率。
原文摘要 · Abstract (English)
Quality-Diversity optimization comprises a family of evolutionary algorithms aimed at generating a collection of diverse and high-performing solutions. MAP-Elites (ME), a notable example, is used effectively in fields like evolutionary robotics. However, the reliance of ME on random mutations from Genetic Algorithms limits its ability to evolve high-dimensional solutions. Methods proposed to overcome this include using gradient-based operators like policy gradients or natural evolution strategies. While successful at scaling ME for neuroevolution, these methods often suffer from slow training speeds, or difficulties in scaling with massive parallelization due to high computational demands or reliance on centralized actor-critic training. In this work, we introduce a fast, sample-efficient ME based algorithm capable of scaling up with massive parallelization, significantly reducing runtimes without compromising performance. Our method, ASCII-ME, unlike existing policy gradient quality-diversity methods, does not rely on centralized actor-critic training. It performs behavioral variations based on time step performance metrics and maps these variations to solutions using policy gradients. Our experiments show that ASCII-ME can generate a diverse collection of high-performing deep neural network policies in less than 250 seconds on a single GPU. Additionally, it operates on average, five times faster than state-of-the-art algorithms while still maintaining competitive sample efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。