用扩散模型实现可灵活调整的机器人行走控制,无需重新训练
Flexible Locomotion Learning with Diffusion Model Predictive Control
- 用扩散模型预测未来状态和动作,结合奖励与约束优化
- 在真实机器人上实现稳定行走,支持测试时动态调整目标
- 通过交互式训练提升泛化能力,适应新任务无需重训
腿式机器人运动需要既鲁棒又灵活的控制器,同时满足任务与安全要求。传统无模型强化学习方法生成固定策略,难以在测试时适应新行为;而经典模型预测控制(MPC)依赖精确动力学模型,难以在复杂环境中获取且常需简化假设。本文提出Diffusion-MPC,利用学习的生成扩散模型作为动力学先验进行规划,通过奖励与约束优化实现测试时灵活行为合成。该方法联合预测未来状态与动作,在每步反向过程中融入奖励规划并施加约束投影,生成满足任务目标且符合物理限制的轨迹。为提升规划模型在模仿预训练外的适应性,引入交互式训练算法:在环境中执行奖励与约束规划器,根据实际回报筛选并重加权收集的轨迹后更新去噪器。实验验证了Diffusion-MPC在真实机器人上的强健行走能力和灵活适应性。
原文摘要 · Abstract (English)
Legged locomotion demands controllers that are both robust and adaptable, while remaining compatible with task and safety considerations. However, model-free reinforcement learning (RL) methods often yield a fixed policy that can be difficult to adapt to new behaviors at test time. In contrast, Model Predictive Control (MPC) provides a natural approach to flexible behavior synthesis by incorporating different objectives and constraints directly into its optimization process. However, classical MPC relies on accurate dynamics models, which are often difficult to obtain in complex environments and typically require simplifying assumptions. We present Diffusion-MPC, which leverages a learned generative diffusion model as an approximate dynamics prior for planning, enabling flexible test-time adaptation through reward and constraint based optimization. Diffusion-MPC jointly predicts future states and actions; at each reverse step, we incorporate reward planning and impose constraint projection, yielding trajectories that satisfy task objectives while remaining within physical limits. To obtain a planning model that adapts beyond imitation pretraining, we introduce an interactive training algorithm for diffusion based planner: we execute our reward-and-constraint planner in environment, then filter and reweight the collected trajectories by their realized returns before updating the denoiser. Our design enables strong test-time adaptability, allowing the planner to adjust to new reward specifications without retraining. We validate Diffusion-MPC on real world, demonstrating strong locomotion and flexible adaptation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。