用LTL逻辑约束让强化学习更安全，有效减少违规行为。
Integrating LTL Constraints into PPO for Safe Reinforcement Learning
- 将LTL安全规则转为惩罚信号，通过拉格朗日方法指导策略优化。
- 在Zones和CARLA环境中，安全违规率显著降低，性能不下降。
- 适合需要严格安全保证的机器人、自动驾驶等场景。
本文提出一种结合线性时序逻辑（LTL）约束的近端策略优化框架（PPO-LTL），用于安全强化学习。LTL可精确表达机器人领域中复杂的安全部署要求，如法规限制。通过极限确定性布尔自动机实时监测违反LTL的行为，并利用逻辑-代价机制将其转化为惩罚信号。这些信号通过拉格朗日方法融入策略优化过程。在Zones与CARLA环境中的大量实验表明，PPO-LTL能持续降低安全违规次数，同时保持与现有先进方法相当的性能表现。代码已开源：https://github.com/EVIEHub/PPO-LTL。
原文摘要 · Abstract (English)
This paper proposes Proximal Policy Optimization with Linear Temporal Logic Constraints (PPO-LTL), a framework that integrates safety constraints written in LTL into PPO for safe reinforcement learning. LTL constraints offer rigorous representations of complex safety requirements, such as regulations that broadly exist in robotics, enabling systematic monitoring of safety requirements. Violations against LTL constraints are monitored by limit-deterministic Büchi automata, and then translated by a logic-to-cost mechanism into penalty signals. The signals are further employed for guiding the policy optimization via the Lagrangian scheme. Extensive experiments on the Zones and CARLA environments show that our PPO-LTL can consistently reduce safety violations, while maintaining competitive performance, against the state-of-the-art methods. The code is at https://github.com/EVIEHub/PPO-LTL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。