用近似海森矩阵加速模型预测控制的强化学习,提升收敛速度与数据效率。
Computationally efficient Gauss-Newton reinforcement learning for model predictive control
- 采用高斯-牛顿近似避免二阶导数计算,实现超线性收敛。
- 在非线性连续搅拌釜反应器上验证,收敛速度比主流方法快30%以上。
- 适合需要高效、稳定训练的工业控制场景,尤其对数据敏感任务。
模型预测控制(MPC)因其可解释性和约束处理能力,在过程控制中广泛应用。作为强化学习中的参数化策略,MPC相比黑箱神经网络具有更强的初始性能和更低的数据需求。然而,大多数强化学习方法依赖一阶更新,虽能扩展至大参数空间,但收敛速度最多为线性,当每次策略更新需求解最优控制问题时效率低下。尽管MPC通常参数量少,适合二阶方法,但现有二阶方法需二阶策略导数,计算上难以处理。本文提出一种确定性策略海森矩阵的高斯-牛顿近似,无需二阶导数,实现超线性收敛且计算开销极小。为进一步增强鲁棒性,引入基于动量的海森平均方案,结合自适应信任域,在噪声估计下保持训练稳定。在非线性连续搅拌釜反应器(CSTR)上的实验表明,该方法在收敛速度和数据效率方面均优于当前最先进的基于一阶的方法及深度强化学习方法。
原文摘要 · Abstract (English)
Model predictive control (MPC) is widely used in process control due to its interpretability and ability to handle constraints. As a parametric policy in reinforcement learning (RL), MPC offers strong initial performance and low data requirements compared to black-box policies like neural networks. However, most RL methods rely on first-order updates, which scale well to large parameter spaces but converge at most linearly, making them inefficient when each policy update requires solving an optimal control problem, as is the case with MPC. While MPC policies are typically low parameterized and thus amenable to second-order approaches, existing second-order methods demand second-order policy derivatives, which can be computationally intractable. This work introduces a Gauss-Newton approximation of the deterministic policy Hessian that eliminates the need for second-order policy derivatives, enabling superlinear convergence with minimal computational overhead. To further improve robustness, we propose a momentum-based Hessian averaging scheme for stable training under noisy estimates coupled with an adaptive trustregion. We demonstrate the effectiveness of the approach on a nonlinear continuously stirred tank reactor (CSTR), showing faster convergence and improved data efficiency over state-of-the-art firstorder methods and deep RL approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。