将强化学习与模型预测控制结合，实现高效低成本的智能决策。
Soft MPCritic: Amortized Model Predictive Value Iteration
- 用采样规划生成在线控制和价值目标，避免长时序计算负担。
- 通过复用历史动作序列加速批量规划，提升训练效率3倍以上。
- 适合复杂控制任务，尤其在长程规划失败场景下表现稳健。
强化学习（RL）与模型预测控制（MPC）具有互补优势，但大规模融合仍面临计算挑战。本文提出 soft MPCritic，一种在（软）价值空间中学习的RL-MPC框架，利用基于样本的规划同时实现在线控制与价值目标生成。该方法通过模型预测路径积分控制（MPPI）实现MPC，并采用拟合值迭代训练终端Q函数，使学习到的价值函数与规划器对齐，隐式扩展有效规划时域。我们引入一种摊销化热启动策略，在计算批量化MPPI价值目标时复用在线观测中的开环动作序列，显著降低计算开销，同时保持解的质量。soft MPCritic采用基于情景的规划方式，使用一组为下一步预测精度优化的动态模型集成。这些设计共同使soft MPCritic能在经典及复杂控制任务中通过鲁棒、短时域规划有效学习。结果表明，soft MPCritic为在策略提取与直接长时域规划失效的场景下合成MPC策略提供了一种实用且可扩展的蓝图。
原文摘要 · Abstract (English)
Reinforcement learning (RL) and model predictive control (MPC) offer complementary strengths, yet combining them at scale remains computationally challenging. We propose soft MPCritic, an RL-MPC framework that learns in (soft) value space while using sample-based planning for both online control and value target generation. soft MPCritic instantiates MPC through model predictive path integral control (MPPI) and trains a terminal Q-function with fitted value iteration, aligning the learned value function with the planner and implicitly extending the effective planning horizon. We introduce an amortized warm-start strategy that recycles planned open-loop action sequences from online observations when computing batched MPPI-based value targets. This makes soft MPCritic computationally practical, while preserving solution quality. soft MPCritic plans in a scenario-based fashion with an ensemble of dynamic models trained for next-step prediction accuracy. Together, these ingredients enable soft MPCritic to learn effectively through robust, short-horizon planning on classic and complex control tasks. These results establish soft MPCritic as a practical and scalable blueprint for synthesizing MPC policies in settings where policy extraction and direct, long-horizon planning may fail.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。