用对抗强化学习训练无人机追逐与躲避,实现真实飞行中的敏捷反应。
Learned Controllers for Agile Quadrotors in Pursuit-Evasion Games
- 采用分阶段异步群体训练,动态选择难对付的对手策略
- 在仿真中实现零样本迁移,真实飞行速度达4.87米/秒
- 生成多样策略组合,提升控制器鲁棒性,适合复杂对抗场景
本文研究1对1四轴无人机追逐-逃避问题,通过强化学习让追逐者与逃避者相互竞争训练。此类对抗场景面临政策随训练变化导致环境非平稳、过拟合当前对手、策略循环或崩溃等挑战。为此提出异步多阶段基于博弈树采样的群体训练方法(AMSPBH),基于策略空间响应原语(PSRO)并适配四轴无人机控制。该方法维护历史策略群体,新策略针对群体混合策略进行近似最优响应训练;每代仅更新一个智能体,使用PPO算法对抗固定对手策略,同时用Hedge采样器提高当前较难应对对手的采样概率。实验表明:(i) AMSPBH能持续发现新策略且保留对旧策略的应对能力,进入收益递减阶段;(ii) 相比仅训练对抗最新对手,群体训练在多样化未见策略上泛化性能更优;(iii) 学习到的策略群体包含明确的追逐与逃避行为,提供战略多样性以发现弱点并增强控制器鲁棒性。通过硬件实验在Crazyflie无刷四轴无人机上验证,实现对人工及学习型对手的零样本仿真到现实迁移,物理飞行最高速度达4.87米/秒。
原文摘要 · Abstract (English)
In this letter we study 1v1 quadrotor pursuit-evasion, where a pursuer and an evader are trained via reinforcement learning (RL) by competing against each other. Such adversarial settings face well-known challenges: each agent's policy changes during training, creating a non-stationary environment; agents might overfit to the current opponent and forget earlier strategies (catastrophic forgetting); and the competitive dynamics can cause strategy cycling or policy collapse. To address these issues, we propose Asynchronous Multi-Stage Population-Based training with Hedge sampling (AMSPBH), a method based on Policy-Space Response Oracles (PSRO) and adapted to quadrotor RL control. PSRO maintains a population of previously trained policies and trains new approximate best responses against mixtures of that population instead of against a single opponent. In AMSPBH, each generation trains one agent with Proximal Policy Optimization (PPO) against frozen opponent policies, while a Hedge sampler assigns higher probability to opponents that are currently difficult to beat. We show that: (i) AMSPBH discovers new strategies while retaining competence against older opponents, reaching a regime where additional best-response training gives limited improvement; (ii) compared to training against only the latest opponent, population-based training generalizes better across diverse and unseen strategies; and (iii) the learned population contains distinct pursuit and evasion behaviors, providing useful strategic diversity for finding weaknesses and improving controller robustness. We validate the trained policies with hardware experiments on Crazyflie brushless quadrotors, showing zero-shot sim-to-real transfer of agile, reactive pursuit-evasion behavior against both handcrafted and learned adversaries, with physical flights reaching up to 4.87 m/s.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。