arXiv:2605.04568cs.LGcs.AI2026-05中稿 · ICML被引 1

用梯度优化潜在想象,提升模型预测控制的效率与性能。

Dream-MPC: Gradient-Based Model Predictive Control with Latent Imagination

论文配图:Dream-MPC: Gradient-Based Model Predictive Control with Latent Imagination
图 1 · 摘自论文原文
  • 基于学习的世界模型,通过梯度上升优化少数候选轨迹。
  • 在24个连续控制任务中显著超越梯度自由方法和当前最优基线。
  • 通过复用历史动作和不确定性正则化,实现高效计算与稳定控制。

当前最先进的基于模型强化学习方法通常采用无梯度、群体式规划、学习策略网络,或两者的结合。融合模型预测控制(MPC)与学习模型及策略先验的混合方法展现出良好前景,但多数依赖无梯度优化,对高维控制任务计算开销大。尽管梯度方法更具潜力,但近期研究显示其性能常劣于无梯度方法。本文提出 Dream-MPC:从滚动策略生成少量候选轨迹,利用学习的世界模型、不确定性正则化,并通过时间上优化迭代的摊销机制复用先前优化动作,以梯度上升方式优化每条轨迹。在24个连续控制任务上的实验表明,Dream-MPC能显著提升底层策略性能,优于无梯度MPC和当前先进基线。代码与视频见 https://dream-mpc.github.io。

原文摘要 · Abstract (English)

State-of-the-art model-based Reinforcement Learning (RL) approaches either use gradient-free, population-based methods for planning, learned policy networks, or a combination of policy networks and planning. Hybrid approaches that combine Model Predictive Control (MPC) with a learned model and a policy prior to leverage the advantages of both paradigms have shown promising results. However, these approaches typically rely on gradient-free optimization methods, which can be computationally expensive for high-dimensional control tasks. While gradient-based methods are a promising alternative, recent works have empirically shown that gradient-based methods often perform worse than their gradient-free counterparts. We propose Dream-MPC, a novel approach that generates few candidate trajectories from a rolled-out policy and optimizes each trajectory by gradient ascent using a learned world model, uncertainty regularization and amortization of optimization iterations over time by reusing previously optimized actions. Our results on 24 continuous control tasks show that Dream-MPC can significantly improve the performance of the underlying policy and can outperform gradient-free MPC and state-of-the-art baselines. Code and videos are available at https://dream-mpc.github.io.

模型预测控制梯度优化强化学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。