arXiv:2412.11764cs.ROcs.LG2024-12中稿 · RA-L被引 23

提出SimpleFlight框架,让无人机强化学习策略无需训练直接在真实飞行中稳定表现。

What Matters in Learning A Zero-Shot Sim-to-Real RL Policy for Quadrotor Control? A Comprehensive Study

  • 融合五项关键技术,基于PPO构建可零样本部署的飞行控制策略。
  • 在真实四旋翼上轨迹跟踪误差降低50%以上,尤其在复杂轨迹上表现更优。
  • 开源模拟器与模型,适合机器人控制、强化学习研究者参考使用。

实现精准敏捷的飞行对四旋翼无人机在各类应用中至关重要。传统控制方法依赖平坦轨迹或耗时优化,灵活性受限。近年来,基于强化学习(RL)的策略因能直接从观测映射到动作,减少对系统知识和执行约束的依赖,成为有前景的替代方案。然而,模拟到现实的差距仍是主要挑战,导致RL策略在真实环境中常出现不稳定。本文系统研究了实现鲁棒零样本部署的五项关键因素,并提出基于PPO的训练框架SimpleFlight,集成这些技术。在Crazyflie四旋翼上验证表明,SimpleFlight相比最先进的RL基线,轨迹跟踪误差减少超过50%。该策略在平滑多项式轨迹及高难度不可行锯齿形轨迹上均表现优异,而基线方法在高速或不可行轨迹下表现不佳。为支持后续研究与复现,我们将SimpleFlight集成至基于GPU的模拟器Omnidrones,并开放代码与模型检查点。项目详情请访问:https://sites.google.com/view/simpleflight/

原文摘要 · Abstract (English)

Executing precise and agile flight maneuvers is critical for quadrotors in various applications. Traditional quadrotor control approaches are limited by their reliance on flat trajectories or time-consuming optimization, which restricts their flexibility. Recently, RL-based policy has emerged as a promising alternative due to its ability to directly map observations to actions, reducing the need for detailed system knowledge and actuation constraints. However, a significant challenge remains in bridging the sim-to-real gap, where RL-based policies often experience instability when deployed in real world. In this paper, we investigate key factors for learning robust RL-based control policies that are capable of zero-shot deployment in real-world quadrotors. We identify five critical factors and we develop a PPO-based training framework named SimpleFlight, which integrates these five techniques. We validate the efficacy of SimpleFlight on Crazyflie quadrotor, demonstrating that it achieves more than a 50% reduction in trajectory tracking error compared to state-of-the-art RL baselines. The policy derived by SimpleFlight consistently excels across both smooth polynominal trajectories and challenging infeasible zigzag trajectories on small thrust-to-weight quadrotors. In contrast, baseline methods struggle with high-speed or infeasible trajectories. To support further research and reproducibility, we integrate SimpleFlight into a GPU-based simulator Omnidrones and provide open-source access to the code and model checkpoints. We hope SimpleFlight will offer valuable insights for advancing RL-based quadrotor control. For more details, visit our project website at https://sites.google.com/view/simpleflight/.

强化学习四旋翼零样本仿真到现实

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。