arXiv:2412.04233cs.LGcs.AI2024-12NeurIPS被引 10

用自适应超网络让多智能体策略自动分化,兼顾效率与多样性。

HyperMARL: Adaptive Hypernetworks for Multi-Agent RL

  • 引入代理条件超网络生成专属参数,解耦观测与身份梯度干扰。
  • 22个场景下性能媲美6种基线,30个智能体时仍保持行为多样性。
  • 无需手动调参或复杂目标设计,适合需要灵活协作的多智能体系统。

多智能体强化学习中的自适应协作需支持统一、专精或混合行为,但实现这种灵活性仍是关键挑战。尽管参数共享能提升学习效率,却会抑制特化所需的行为多样性,其根源在于跨智能体梯度干扰。我们发现,将智能体编号与观测耦合会显著加剧此问题。现有方法通常通过修改目标函数、人工设定多样性水平或顺序更新来缓解,但这增加了复杂性。我们提出一种新思路:利用代理条件超网络生成专属参数,直接解耦观测与代理相关的梯度,从而消除耦合带来的干扰。所提方法HyperMARL无需复杂设计,实验证明其有效降低策略梯度方差。在22个不同基准任务(最多30个智能体)中,其性能可与六种主流基线媲美,同时保持接近非参数共享方法的行为多样性,展现出通用且原理清晰的适应性多智能体学习能力。代码已开源。

原文摘要 · Abstract (English)

Adaptive cooperation in multi-agent reinforcement learning (MARL) requires policies to express homogeneous, specialised, or mixed behaviours, yet achieving this adaptivity remains a critical challenge. While parameter sharing (PS) is standard for efficient learning, it notoriously suppresses the behavioural diversity required for specialisation. This failure is largely due to cross-agent gradient interference, a problem we find is surprisingly exacerbated by the common practice of coupling agent IDs with observations. Existing remedies typically add complexity through altered objectives, manual preset diversity levels, or sequential updates -- raising a fundamental question: can shared policies adapt without these intricacies? We propose a solution built on a key insight: an agent-conditioned hypernetwork can generate agent-specific parameters and decouple observation- and agent-conditioned gradients, directly countering the interference from coupling agent IDs with observations. Our resulting method, HyperMARL, avoids the complexities of prior work and empirically reduces policy gradient variance. Across diverse MARL benchmarks (22 scenarios, up to 30 agents), HyperMARL achieves performance competitive with six key baselines while preserving behavioural diversity comparable to non-parameter sharing methods, establishing it as a versatile and principled approach for adaptive MARL. The code is publicly available at https://github.com/KaleabTessera/HyperMARL.

多智能体强化学习自适应超网络

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。