arXiv:2410.15979cs.RO2024-10ICRA被引 30

用可微仿真让无人机从视觉中快速学会控制,效率远超传统强化学习。

Learning Quadrotor Control From Visual Features Using Differentiable Simulation

  • 通过可微仿真计算梯度,实现高效策略优化。
  • 仅用视觉特征时,1分钟内完成无人机恢复控制训练。
  • 适合做视觉感知机器人控制的快速原型开发。

强化学习在机器人领域仍面临样本效率低的挑战,需大量模拟且训练时间长。尤其在依赖视觉的状态估计任务中,问题更为突出。可微仿真通过动态模型反向传播梯度,提供低方差的解析梯度,从而提升样本效率。本文展示了其在四旋翼无人机控制中的巨大潜力:使用可微仿真训练相比无模型强化学习,在样本效率和训练时间上均有显著优势。当输入包含飞行器状态时,策略可在数秒内学会恢复;仅依赖视觉特征时,也仅需数分钟即可完成训练。成功关键在于:第一,采用简单代理模型进行梯度计算,大幅加速训练而无需牺牲性能;第二,结合状态表示学习与策略学习,提升了纯视觉任务下的收敛速度。这些结果表明,可微仿真为真实机器人任务提供了极具前景的替代方案。

原文摘要 · Abstract (English)

The sample inefficiency of reinforcement learning (RL) remains a significant challenge in robotics. RL requires large-scale simulation and can still cause long training times, slowing research and innovation. This issue is particularly pronounced in vision-based control tasks where reliable state estimates are not accessible. Differentiable simulation offers an alternative by enabling gradient back-propagation through the dynamics model, providing low-variance analytical policy gradients and, hence, higher sample efficiency. However, its usage for real-world robotic tasks has yet been limited. This work demonstrates the great potential of differentiable simulation for learning quadrotor control. We show that training in differentiable simulation significantly outperforms model-free RL in terms of both sample efficiency and training time, allowing a policy to learn to recover a quadrotor in seconds when providing vehicle states and in minutes when relying solely on visual features. The key to our success is two-fold. First, the use of a simple surrogate model for gradient computation greatly accelerates training without sacrificing control performance. Second, combining state representation learning with policy learning enhances convergence speed in tasks where only visual features are observable. These findings highlight the potential of differentiable simulation for real-world robotics and offer a compelling alternative to conventional RL approaches.

无人机控制可微仿真强化学习视觉感知

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。