动态修复逻辑规范，让强化学习更安全且不丢目标。
Adaptive GR(1) Specification Repair for Liveness-Preserving Shielding in Reinforcement Learning
- 用可自适应的GR(1)规范在线修复环境异常时的安全约束。
- 在Minepump和Atari Seaquest上实现近最优奖励与完全合规。
- 适合需持续安全保障的复杂决策场景，如自动驾驶、机器人控制。
屏蔽技术广泛用于强化学习中的安全约束，确保智能体行为符合形式化规范。传统屏蔽方法多为静态，依赖预设逻辑规范和手工抽象，当环境假设被违反时无法应对。本文提出基于广义反应性第1阶（GR(1)）规范的自适应屏蔽框架，该规范是线性时序逻辑（LTL）中可计算且表达力强的片段，能同时刻画安全性和活性属性。方法在运行时检测环境假设违背，并通过归纳逻辑编程（ILP）自动在线修复GR(1)规范，系统且可解释地实现规范演化，确保活性可达，仅在必要时最小化削弱目标。两个案例研究（Minepump和Atari Seaquest）表明：（i）静态符号控制器在优化辅助奖励时表现严重次优；（ii）配备本方法的强化学习智能体，在保持近似最优奖励的同时实现完美逻辑合规，优于静态屏蔽方案。
原文摘要 · Abstract (English)
Shielding is widely used to enforce safety in reinforcement learning (RL), ensuring that an agent's actions remain compliant with formal specifications. Classical shielding approaches, however, are often static, in the sense that they assume fixed logical specifications and hand-crafted abstractions. While these static shields provide safety under nominal assumptions, they fail to adapt when environment assumptions are violated. In this paper, we develop an adaptive shielding framework based on based on Generalized Reactivity of rank 1 (GR(1)) specifications, a tractable and expressive fragment of Linear Temporal Logic (LTL) that captures both safety and liveness properties. Our method detects environment assumption violations at runtime and employs Inductive Logic Programming (ILP) to automatically repair GR(1) specifications online, in a systematic and interpretable way. This ensures that the shield evolves gracefully, ensuring liveness is achievable and minimally weakening goals only when necessary. We consider two case studies: Minepump and Atari Seaquest; showing that (i) static symbolic controllers are often severely suboptimal when optimizing for auxiliary rewards, and (ii) RL agents equipped with our adaptive shield maintain near-optimal reward and perfect logical compliance compared with static shields.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。