arXiv:2410.15127cs.LGcs.AI2024-10

提出可验证可解释的安全强化学习框架,确保模型满足预设约束。

Reinfier and Reintrainer: Verification and Interpretation-Driven Safe Deep Reinforcement Learning Frameworks

  • 通过形式化验证实时检测模型与约束的差距
  • 在六项基准上同时提升性能与约束满足率
  • 适合需要安全可信AI的工业级应用

确保深度强化学习(DRL)的可验证性和可解释性安全对其实现真实应用至关重要。现有方法如闭环验证训练面临部署困难、训练效率低、缺乏可解释性及属性满足率和奖励表现不佳等问题。本文提出新型验证驱动、可解释闭环框架Reintrainer,用于构建可信的DRL模型,确保其满足预期约束属性。具体而言,每轮迭代中,该框架使用形式化验证测量训练中模型与预设属性之间的差距,分析各输入特征对输出的贡献,并基于实时测量结果生成训练策略,直至所有预设属性被证明成立。此外,为解决现有验证器与解释器复用性差的问题,我们开发了Reinfier——一个嵌入Reintrainer的通用基础工具,具备断点搜索与验证驱动解释功能,配套简洁的约束编码语言DRLP。评估表明,Reintrainer在六个公开基准上均优于当前最先进方法,在性能和属性保障方面表现更优。框架开源地址:https://github.com/Kurayuri/Reinfier。

原文摘要 · Abstract (English)

Ensuring verifiable and interpretable safety of deep reinforcement learning (DRL) is crucial for its deployment in real-world applications. Existing approaches like verification-in-the-loop training, however, face challenges such as difficulty in deployment, inefficient training, lack of interpretability, and suboptimal performance in property satisfaction and reward performance. In this work, we propose a novel verification-driven interpretation-in-the-loop framework Reintrainer to develop trustworthy DRL models, which are guaranteed to meet the expected constraint properties. Specifically, in each iteration, this framework measures the gap between the on-training model and predefined properties using formal verification, interprets the contribution of each input feature to the model's output, and then generates the training strategy derived from the on-the-fly measure results, until all predefined properties are proven. Additionally, the low reusability of existing verifiers and interpreters motivates us to develop Reinfier, a general and fundamental tool within Reintrainer for DRL verification and interpretation. Reinfier features breakpoints searching and verification-driven interpretation, associated with a concise constraint-encoding language DRLP. Evaluations demonstrate that Reintrainer outperforms the state-of-the-art on six public benchmarks in both performance and property guarantees. Our framework can be accessed at https://github.com/Kurayuri/Reinfier.

强化学习安全验证可解释性形式化方法

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。