用可凸优化的神经网络修正离线强化学习动作,提升化工反应器控制的安全性与稳定性。
Safe Deployment of Offline Reinforcement Learning via Input Convex Action Correction
- 部署时引入基于输入凸神经网络的动作修正层,实时优化策略输出。
- 结合修正后的方法在三种工况下均超越传统控制,保持系统稳定。
- 适用于高风险工业过程,适合关注安全可控的自动化研究者。
离线强化学习为利用历史数据开发化学过程控制策略提供了可行框架,无需在线实验带来的风险与成本。本文将离线RL应用于间歇式放热聚合反应器的高效安全控制,构建了兼容Gymnasium的仿真环境,真实模拟反应动力学、能量平衡及操作约束。该环境涵盖启动、降等级和升等级三种工业常见场景,并提供由比例积分控制器随机调参生成的可复现离线数据集,作为评估算法的基准。评估行为克隆与隐式Q学习等基线方法,发现其存在稳态偏差和设定点附近性能下降问题。为此,提出一种部署时的安全层,通过输入凸神经网络(PICNN)作为学习成本模型,进行梯度驱动的动作修正。该方法可在不重新训练或与环境交互的前提下,实时、可微地沿状态相关凸成本面下降,实现安全动作调整。实验表明,结合凸动作修正的离线RL在所有场景中均优于传统控制,且保持系统稳定。结果证明了将离线RL与可解释、安全感知修正相结合在高风险化工控制中的可行性,为工业数据驱动自动化奠定了可靠基础。
原文摘要 · Abstract (English)
Offline reinforcement learning (offline RL) offers a promising framework for developing control strategies in chemical process systems using historical data, without the risks or costs of online experimentation. This work investigates the application of offline RL to the safe and efficient control of an exothermic polymerisation continuous stirred-tank reactor. We introduce a Gymnasium-compatible simulation environment that captures the reactor's nonlinear dynamics, including reaction kinetics, energy balances, and operational constraints. The environment supports three industrially relevant scenarios: startup, grade change down, and grade change up. It also includes reproducible offline datasets generated from proportional-integral controllers with randomised tunings, providing a benchmark for evaluating offline RL algorithms in realistic process control tasks. We assess behaviour cloning and implicit Q-learning as baseline algorithms, highlighting the challenges offline agents face, including steady-state offsets and degraded performance near setpoints. To address these issues, we propose a novel deployment-time safety layer that performs gradient-based action correction using input convex neural networks (PICNNs) as learned cost models. The PICNN enables real-time, differentiable correction of policy actions by descending a convex, state-conditioned cost surface, without requiring retraining or environment interaction. Experimental results show that offline RL, particularly when combined with convex action correction, can outperform traditional control approaches and maintain stability across all scenarios. These findings demonstrate the feasibility of integrating offline RL with interpretable and safety-aware corrections for high-stakes chemical process control, and lay the groundwork for more reliable data-driven automation in industrial systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。