将探索与安全解耦,实现高效又安全的强化学习。
Guardian: Decoupling Exploration from Safety in Reinforcement Learning
- 用独立探索器和投影守护者分离策略优化与安全约束
- Atari-100k上得分3.02,比之前方法高45%且更稳定
- 适合需要安全探索的长时序或高风险任务
混合离线-在线强化学习(O2O RL)兼具样本效率和鲁棒探索潜力,但因离线与在线数据分布偏移导致训练不稳定。本文提出RLPD-GX框架,将策略优化与安全约束解耦:奖励驱动的学习器自由探索,基于投影的守护者确保规则一致执行和安全价值回溯。该设计保留在线交互的探索优势,避免陷入保守策略。为进一步稳定训练,引入动态课程,逐步扩展时间范围并调节离线-在线数据混合比例。理论上证明了受保护贝尔曼算子的收缩性,实证显示在Atari-100k上取得3.02的归一化均分,较前代方法提升45%,同时具备更强安全性和稳定性。跨任务消融实验表明,该方法在安全关键及长时序任务中均有显著提升,验证了其普适性。结果表明,解耦安全机制是实现鲁棒O2O RL的简单而严谨路径,为探索与安全的平衡提供新范式。
原文摘要 · Abstract (English)
Hybrid offline--online reinforcement learning (O2O RL) promises both sample efficiency and robust exploration, but suffers from instability due to distribution shift between offline and online data. We introduce RLPD-GX, a framework that decouples policy optimization from safety enforcement: a reward-seeking learner explores freely, while a projection-based guardian guarantees rule-consistent execution and safe value backups. This design preserves the exploratory value of online interactions without collapsing to conservative policies. To further stabilize training, we propose dynamic curricula that gradually extend temporal horizons and anneal offline--online data mixing. We prove convergence via a contraction property of the guarded Bellman operator, and empirically show state-of-the-art performance on Atari-100k, achieving a normalized mean score of 3.02 (+45\% over prior hybrid methods) with stronger safety and stability. Beyond Atari, ablations demonstrate consistent gains across safety-critical and long-horizon tasks, underscoring the generality of our design. Extensive and comprehensive results highlight decoupled safety enforcement as a simple yet principled route to robust O2O RL, suggesting a broader paradigm for reconciling exploration and safety in reinforcement learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。