用可解释神经网络初始化强化学习,提升安全控制性能。
Reinforcement Learning-based Control via Y-wise Affine Neural Networks (YANNs)
- 用YANN网络精确表示线性最优控制解,作为策略起点
- 在非线性系统上在线训练,最终逼近复杂非线性控制问题解
- 特别适合对安全性要求高的系统,优于传统深度强化学习
本文提出基于Y-wise Affine Neural Networks(YANNs)的新型强化学习(RL)算法。YANNs 是一种可解释的神经网络,能够精确表示任意输入输出维度、定义在多个多面体子域上的分段仿射函数。其典型应用是重构多参数线性模型预测控制的显式解。在此基础上,我们利用YANNs初始化RL的策略网络和价值网络,使YANN-RL算法从线性最优控制的确定性出发。YANN-策略通过离线计算的近似线性系统模型获得的多参数控制解进行初始化;YANN-价值函数则表示线性系统的状态-动作值函数及最优控制问题中的奖励函数。额外引入网络层以扩展至非线性表达,并可直接与真实复杂非线性系统在线交互训练。由此,策略与价值函数初始时精确对应线性最优控制,最终能学习一般非线性最优控制问题的解。连续策略优化机制提供启发式保证:线性控制解为强化学习策略性能的有效下界。该算法在剪切摆和安全关键的化学反应系统上进行了验证,结果表明,相较于使用深度确定性策略梯度的现代强化学习算法,YANN-RL在考虑安全约束时表现显著更优。
原文摘要 · Abstract (English)
This work presents a novel reinforcement learning (RL) algorithm based on Y-wise Affine Neural Networks (YANNs). YANNs provide an interpretable neural network which can exactly represent known piecewise affine functions of arbitrary input and output dimensions defined on any amount of polytopic subdomains. One representative application of YANNs is to reformulate explicit solutions of multi-parametric linear model predictive control. Built on this, we propose the use of YANNs to initialize RL actor and critic networks, which enables the resulting YANN-RL control algorithm to start with the confidence of linear optimal control. The YANN-actor is initialized by representing the multi-parametric control solutions obtained via offline computation using an approximated linear system model. The YANN-critic represents the explicit form of the state-action value function for the linear system and the reward function as the objective in an optimal control problem (OCP). Additional network layers are injected to extend YANNs for nonlinear expressions, which can be trained online by directly interacting with the true complex nonlinear system. In this way, both the policy and state-value functions exactly represent a linear OCP initially and are able to eventually learn the solution of a general nonlinear OCP. Continuous policy improvement is also implemented to provide heuristic confidence that the linear OCP solution serves as an effective lower bound to the performance of RL policy. The YANN-RL algorithm is demonstrated on a clipped pendulum and a safety-critical chemical-reactive system. Our results show that YANN-RL significantly outperforms the modern RL algorithm using deep deterministic policy gradient, especially when considering safety constraints.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。