将强化学习与并行计算的模型预测控制结合，提升机器人运动控制的适应性与效率。
Residual MPC: Blending Reinforcement Learning with GPU-Parallelized Model Predictive Control
- 在扭矩层面融合MPC与RL输出，实现可控修正。
- 训练时每秒100次并行评估，样本效率更高，奖励收敛更好。
- 适合需要高鲁棒性与可解释性的机器人控制场景。
模型预测控制（MPC）提供基于物理模型的可解释、可调运动控制器，但其鲁棒性依赖频繁重规划，受限于模型偏差和实时计算能力。强化学习（RL）虽可通过随机训练生成高度鲁棒的行为，却常缺乏可解释性，易出现分布外失败，且需大量奖励工程。本文提出一种GPU并行化的残差架构，通过在扭矩控制层融合MPC与RL输出，实现紧密集成。我们构建了用于强化学习训练的运动学动力学全身体型MPC公式，在数千个代理上以100 Hz并行运行。残差策略学习对MPC输出进行精准修正，结合了模型控制的可解释性与约束处理能力，以及强化学习的适应性。基于模型的控制先验作为强偏置，以简单奖励集引导策略朝理想行为演进。相比独立的MPC或端到端RL，该方法具备更高样本效率，收敛至更优渐近奖励，扩展了可跟踪速度指令范围，并实现了对未见过步态和不平地形的零样本适应。
原文摘要 · Abstract (English)
Model Predictive Control (MPC) provides interpretable, tunable locomotion controllers grounded in physical models, but its robustness depends on frequent replanning and is limited by model mismatch and real-time computational constraints. Reinforcement Learning (RL), by contrast, can produce highly robust behaviors through stochastic training but often lacks interpretability, suffers from out-of-distribution failures, and requires intensive reward engineering. This work presents a GPU-parallelized residual architecture that tightly integrates MPC and RL by blending their outputs at the torque-control level. We develop a kinodynamic whole-body MPC formulation evaluated across thousands of agents in parallel at 100 Hz for RL training. The residual policy learns to make targeted corrections to the MPC outputs, combining the interpretability and constraint handling of model-based control with the adaptability of RL. The model-based control prior acts as a strong bias, initializing and guiding the policy towards desirable behavior with a simple set of rewards. Compared to standalone MPC or end-to-end RL, our approach achieves higher sample efficiency, converges to greater asymptotic rewards, expands the range of trackable velocity commands, and enables zero-shot adaptation to unseen gaits and uneven terrain.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。