用模型预测控制注入行为,让机器人跑步更自然。
MPC-Injection: Biasing Off-Policy Locomotion RL Toward Controller-Induced Behavior Basins

- 通过向经验回放缓冲区注入控制器生成的状态转移,引导强化学习
- 仅用1-2个奖励项就实现与21个调优项相当的自然步态
- 无需修改奖励函数或添加判别器,适合希望快速获得稳定步态的研究者
用于行走的强化学习常收敛到局部最优但无法部署的行为,如肢体抖动或躯干滑行,这些行为虽能最大化回报却无法生成可用步态。我们提出MPC-Injection,一种低开销方法:将求解相同马尔可夫决策过程的模型预测控制器生成的状态转移注入经验回放缓冲区,从而引导强化学习朝设计者偏好的步态方向演进。相比奖励塑造,无需重设计任务奖励;相比对抗式模仿学习,无需判别器、无需运动重定向、也无辅助目标。控制器偏好行为仅通过回放缓冲区的状态分布传递给策略。在仿真2D行走者及真实Go2四足机器人上,我们证明MPC-Injection仅用1-2个任务奖励项即可使策略进入控制器行为域,生成步态质量与使用21个调优项的奖励塑造和去判别器、无重定向的对抗性运动先验相当。进一步分析表明,注入状态转移会引导演员-评论家更新偏向控制器访问过的状态,使策略学习到纯强化学习在简单奖励下难以达到的行为。
原文摘要 · Abstract (English)
Reinforcement learning (RL) for locomotion frequently converges to locally optimal but undeployable behaviors, such as vibrating limbs or scooting on the torso, that maximize return without producing a usable gait. We present MPC-Injection, a low-overhead method that steers RL toward a designer-preferred gait by inserting transitions into the replay buffer from a model predictive controller solving the same Markov decision process. Unlike reward shaping, MPC-Injection does not require redesigning the task reward, and unlike adversarial imitation learning, it adds no discriminator, no kinematic retargeting, and no auxiliary objective. Instead, the controller's preferred behavior is transferred to the policy purely through the replay state distribution. On a 2D walker in simulation and with sim-to-real evaluation on a Go2 quadruped, we show that MPC-Injection drives the policy into the controller's behavior basin using a one to two-term task reward, producing gaits qualitatively comparable to those of reward shaping with twenty-one tuned terms and of adversarial motion priors without their discriminator and retargeting overhead. We further analyze how the injected transitions bias actor-critic updates toward controller-visited states, allowing the policy to learn behaviors that pure RL may fail to reach under simple reward functions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。