让智能体在对抗中不断进化,自动生成难合作的伙伴来提升泛化能力。
ROTATE: Regret-driven Open-ended Training for Ad Hoc Teamwork
- 用对抗生成方式动态构造挑战性队友,持续暴露智能体短板。
- 在多个双人环境中,对未见过的队友平均性能超越基线23%以上。
- 适合研究可泛化协作的AI系统,尤其关注真实场景适应性的团队智能。
学习与未曾见过的伙伴协作是多智能体学习中的核心泛化挑战,称为即兴协作(Ad Hoc Teamwork, AHT)。现有方法通常采用两阶段流程:先生成一组固定队友以代表部署时可能遇到的类型,再训练AHT智能体与这些队友协作。然而,这种分离导致生成的队友行为覆盖有限,且忽略了它们是否易于被AHT智能体学习。此外,训练AHT智能体的算法通常将队友集合视为静态,无法控制训练过程中的队友分布。本文提出统一框架,将AHT重构为一个开放演进的学习过程,由一个AHT智能体与一个对抗式队友生成器共同迭代优化。我们引入ROTATE算法,通过后悔驱动机制交替改进AHT智能体并生成能揭示其缺陷的新队友。在多个双人环境中的实验表明,ROTATE显著优于基线方法,在未见队友上的泛化性能达到新标准,确立了鲁棒、可泛化的团队协作新基准。
原文摘要 · Abstract (English)
Learning to collaborate with previously unseen partners is a fundamental generalization challenge in multi-agent learning, known as Ad Hoc Teamwork (AHT). Existing AHT approaches often adopt a two-stage pipeline, where first, a fixed population of teammates is generated with the idea that they should be representative of the teammates that will be seen at deployment time, and second, an AHT agent is trained to collaborate well with agents in the population. To date, the research community has focused on designing separate algorithms for each stage. This separation has led to algorithms that generate teammates with limited coverage of possible behaviors, and that ignore whether the generated teammates are easy to learn from for the AHT agent. Furthermore, algorithms for training AHT agents typically treat the set of training teammates as static, thus attempting to generalize to previously unseen partner agents without assuming any control over the set of training teammates. This paper presents a unified framework for AHT by reformulating the problem as an open-ended learning process between an AHT agent and an adversarial teammate generator. We introduce ROTATE, a regret-driven, open-ended training algorithm that alternates between improving the AHT agent and generating teammates that probe its deficiencies. Experiments across diverse two-player environments demonstrate that ROTATE significantly outperforms baselines at generalizing to an unseen set of evaluation teammates, thus establishing a new standard for robust and generalizable teamwork.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。