通过分层共自演机制，让三架无人机在排球比赛中实现高效协作与敏捷控制。
Mastering Multi-Drone Volleyball through Hierarchical Co-Self-Play Reinforcement Learning
- 分层设计：高层策略与底层控制分离，提升决策与执行效率。
- 训练出82.9%胜率，超越非分层与两阶段基线模型。
- 无需专家示范，自动涌现角色切换与团队阵型等协同行为。
本文研究3对3多无人机排球任务，这是一个需要高阶战略协调与低阶敏捷控制的具身竞争任务。该任务为回合制、多智能体且物理真实，面临长时序依赖、强智能体耦合及四旋翼欠驱动动力学等挑战。为此，我们提出分层共自演（HCSP）框架，将集中式高层战略决策与分布式底层运动控制解耦。设计三阶段基于种群的训练流程：(I) 训练多样化底层技能；(II) 固定底层技能下通过自演学习高层策略；(III) 通过共自演进行联合微调。实验表明，HCSP平均胜率达82.9%，高于非分层自演与两阶段变体的71.5%。共自演还催生了角色切换与协同阵型等涌现团队行为，验证了分层设计与训练方案的有效性。
原文摘要 · Abstract (English)
In this paper, we tackle the problem of learning to play 3v3 multi-drone volleyball, a new embodied competitive task that requires both high-level strategic coordination and low-level agile control. The task is turn-based, multi-agent, and physically grounded, posing significant challenges due to its long-horizon dependencies, tight inter-agent coupling, and the underactuated dynamics of quadrotors. To address this, we propose Hierarchical Co-Self-Play (HCSP), a hierarchical reinforcement learning framework that separates centralized high-level strategic decision-making from decentralized low-level motion control. We design a three-stage population-based training pipeline to enable both strategy and skill to emerge from scratch without expert demonstrations: (I) training diverse low-level skills, (II) learning high-level strategy via self-play with fixed low-level skills, and (III) joint fine-tuning through co-self-play. Experiments show that HCSP achieves superior performance, outperforming non-hierarchical self-play and rule-based hierarchical baselines with an average 82.9% win rate and a 71.5% win rate against the two-stage variant. Moreover, co-self-play leads to emergent team behaviors such as role switching and coordinated formations, demonstrating the effectiveness of our hierarchical design and training scheme. The project page is at https://hi-co-self-play.github.io.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。