用MPC指导强化学习，让机器人跑得稳、抓得准
Accelerating and Scaling MPC-Guided Reinforcement Learning for Humanoid Locomotion and Manipulation
- 训练时用重心动力学的MPC轨迹生成奖励信号
- 自研并行求解器实现毫秒级批量计算，无需预构建
- 在真实机器人上验证，运动与操作性能显著提升
在人形机器人运动控制中，模型预测控制（MPC）提供物理合理的预测与约束处理，而强化学习（RL）则通过大规模仿真实现鲁棒的全身技能。然而，将MPC嵌入强化学习常需耗时的问题构建或过高训练开销，难以实际应用。本文研究了高效训练期MPC引导的人形机器人运动与操作方法，称为MPC-RL。提出基于重心动力学的MPC奖励公式，在训练阶段利用MPC轨迹提供指导。为支持大规模并行强化学习，开发了$π^n$MPC——一种无需预构建、可在时间变化动力学上直接运行的并行化批处理GPU求解器，有效降低内存占用与编译开销。通过多种对比实验与硬件验证，MPC-RL在运动与操作任务中均表现出更优性能。代码已开源：https://github.com/junhengl/mpc-rl。
原文摘要 · Abstract (English)
In humanoid motion control, model predictive control (MPC) offers physically grounded prediction and constraint handling, while reinforcement learning (RL) enables robust whole-body skills through large-scale simulation. However, using MPC inside RL often requires time-consuming problem construction or excessive training overhead, making such frameworks difficult to justify in practice. This work studies efficient training-time MPC guidance for humanoid locomotion and manipulation, termed MPC-RL. We introduce a centroidal-dynamics MPC reward formulation that leverages guidance from MPC trajectories in training time. To make this practical in massively parallel RL, we develop $π^n$MPC, a parallel-in-horizon and construction-free batched GPU MPC solver that operates directly on time-varying dynamics to avoid high memory usage and pre-compilation. Through a variety of comparative studies and hardware validations, we have found that MPC-RL achieves superior performance in locomotion and manipulation skills. The code base is available at https://github.com/junhengl/mpc-rl.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。