用分阶段训练让四旋翼快速稳定,省时省资源。
Curriculum-based Sample Efficient Reinforcement Learning for Robust Stabilization of a Quadrotor
- 分三阶段逐步提升任务难度,每阶段传递学习成果。
- 仅用1/4样本量即达传统方法性能,收敛快3倍以上。
- 适合需要快速部署的无人机稳定控制场景。
本文提出一种新型样本高效的课程学习(CL)方法,用于训练端到端强化学习策略,实现四旋翼在随机初始条件下的位置与航向稳定控制(直接控制电机转速),同时满足预设的瞬态与稳态性能要求。该目标在空中巡检中至关重要,但传统单阶段端到端强化学习需大量计算资源和长时间训练。为此,本文受人类学习启发,将任务分解为三个逐步增加复杂度的阶段:悬停、平移与旋转自由度耦合、对随机非零初速度的鲁棒性。每个阶段采用定制奖励函数与截断条件,实现知识迁移。实验表明,该方法在相同奖励函数与超参数下,性能优于单阶段训练策略,且计算资源需求显著降低(样本量减少约75%),收敛时间缩短约67%。在Gym-PyBullet-Drones仿真环境中,针对随机初始条件及巡检位姿跟踪场景进行了充分验证。视频演示见https://youtu.be/9wv6T4eezAU。
原文摘要 · Abstract (English)
This article introduces a novel sample-efficient curriculum learning (CL) approach for training an end-to-end reinforcement learning (RL) policy for robust stabilization of a Quadrotor. The learning objective is to simultaneously stabilize position and yaw-orientation from random initial conditions through direct control over motor RPMs (end-to-end), while adhering to pre-specified transient and steady-state specifications. This objective, relevant in aerial inspection applications, is challenging for conventional one-stage end-to-end RL, which requires substantial computational resources and lengthy training times. To address this challenge, this article draws inspiration from human-inspired curriculum learning and decomposes the learning objective into a three-stage curriculum that incrementally increases task complexity, while transferring knowledge from one stage to the next. In the proposed curriculum, the policy sequentially learns hovering, the coupling between translational and rotational degrees of freedom, and robustness to random non-zero initial velocities, utilizing a custom reward function and episode truncation conditions. The results demonstrate that the proposed CL approach achieves superior performance compared to a policy trained conventionally in one stage, with the same reward function and hyperparameters, while significantly reducing computational resource needs (samples) and convergence time. The CL-trained policy's performance and robustness are thoroughly validated in a simulation engine (Gym-PyBullet-Drones), under random initial conditions, and in an inspection pose-tracking scenario. A video presenting our results is available at https://youtu.be/9wv6T4eezAU.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。