用对称性结构提升多智能体强化学习的样本效率和泛化能力
Boosting Sample Efficiency and Generalization in Multi-agent Reinforcement Learning via Equivariance
- 引入等变图神经网络增强模型对称性，改善学习结构
- 新方法在通用性测试中比普通GNN提升2到5倍性能
- 适合需要高效训练和强泛化的复杂多智能体系统
多智能体强化学习（MARL）面临样本效率低和泛化能力差的问题，部分源于传统神经网络缺乏结构或归纳偏置。多智能体场景中常见对称性，几何深度学习中的等变图神经网络（EGNN）能对节点的旋转、平移和反射保持对称性，已被证明可提升学习效率并降低误差。本文表明，将EGNN应用于MARL可提升样本效率与泛化能力。但发现直接使用EGNN会导致早期探索偏差。为此提出改进方法E2GN2，通过增强探索能力缓解该问题。在MPE和SMACv2基准测试中，相比其他常见函数逼近器，E2GN2在样本效率、最终奖励收敛性上均有显著提升，并在泛化测试中实现2至5倍性能增益。
原文摘要 · Abstract (English)
Multi-Agent Reinforcement Learning (MARL) struggles with sample inefficiency and poor generalization [1]. These challenges are partially due to a lack of structure or inductive bias in the neural networks typically used in learning the policy. One such form of structure that is commonly observed in multi-agent scenarios is symmetry. The field of Geometric Deep Learning has developed Equivariant Graph Neural Networks (EGNN) that are equivariant (or symmetric) to rotations, translations, and reflections of nodes. Incorporating equivariance has been shown to improve learning efficiency and decrease error [ 2 ]. In this paper, we demonstrate that EGNNs improve the sample efficiency and generalization in MARL. However, we also show that a naive application of EGNNs to MARL results in poor early exploration due to a bias in the EGNN structure. To mitigate this bias, we present Exploration-enhanced Equivariant Graph Neural Networks or E2GN2. We compare E2GN2 to other common function approximators using common MARL benchmarks MPE and SMACv2. E2GN2 demonstrates a significant improvement in sample efficiency, greater final reward convergence, and a 2x-5x gain in over standard GNNs in our generalization tests. These results pave the way for more reliable and effective solutions in complex multi-agent systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。