arXiv:2506.03469cs.AIcs.LG2025-06被引 1

用可解释抽象+风险引导伪造,提升强化学习安全验证的覆盖率和可信度。

Verification-Guided Falsification for Safe RL via Explainable Abstraction and Risk-Aware Exploration

  • 通过可理解抽象生成政策图,供模型检测器验证时序安全规范
  • 若未发现违规,基于风险估计优先搜索低覆盖高危状态区域
  • 提供概率保证并部署轻量级安全盾牌,实时应对潜在风险

在高风险环境中确保强化学习策略的安全性,不仅需要形式化验证,还需可解释性和针对性的伪造测试。虽然模型检测能提供形式保证,但其效果受限于抽象质量和轨迹数据集的完整性。本文提出一种融合(1)可解释性、(2)模型检测、(3)风险引导伪造的混合框架,实现严谨性与覆盖性的平衡。首先利用离线轨迹构建人类可理解的策略抽象图(CAPS),该图既适配模型检测器,又语义清晰,可输入至Storm概率模型检测器以验证时序安全规范。若检测到违规,将返回可解释的反例轨迹;若未发现违规,则因抽象或数据覆盖不足无法断言满足性。此时,我们基于模型检测过程中的风险估计,引导伪造策略优先探索高风险状态及数据稀疏区域,并提供类似PAC的未检出违规概率保证。最后引入轻量级安全盾,在运行时当风险超过阈值时切换至备用策略,实现故障缓解而无需重新训练。

原文摘要 · Abstract (English)

Ensuring the safety of reinforcement learning (RL) policies in high-stakes environments requires not only formal verification but also interpretability and targeted falsification. While model checking provides formal guarantees, its effectiveness is limited by abstraction quality and the completeness of the underlying trajectory dataset. We propose a hybrid framework that integrates (1) explainability, (2) model checking, and (3) risk-guided falsification to achieve both rigor and coverage. Our approach begins by constructing a human-interpretable abstraction of the RL policy using Comprehensible Abstract Policy Summarization (CAPS). This abstract graph, derived from offline trajectories, is both verifier-friendly, semantically meaningful, and can be used as input to Storm probabilistic model checker to verify satisfaction of temporal safety specifications. If the model checker identifies a violation, it will return an interpretable counterexample trace by which the policy fails the safety requirement. However, if no violation is detected, we cannot conclude satisfaction due to potential limitation in the abstraction and coverage of the offline dataset. In such cases, we estimate associated risk during model checking to guide a falsification strategy that prioritizes searching in high-risk states and regions underrepresented in the trajectory dataset. We further provide PAC-style guarantees on the likelihood of uncovering undetected violations. Finally, we incorporate a lightweight safety shield that switches to a fallback policy at runtime when such a risk exceeds a threshold, facilitating failure mitigation without retraining.

强化学习形式化验证可解释性安全控制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。