用分层策略与模型想象，让强化学习在长任务中更安全高效。
Imagine to Ensure Safety in Hierarchical Reinforcement Learning
- 分层设计：高层生成安全子目标，低层通过模型模拟减少危险行为。
- 实验显示：成功率和安全约束满足率显著优于现有方法，跨种子稳定达标。
- 适合研究长时序、高维动作的强化学习安全问题，如机器人导航与操作。
本文研究强化学习中的安全探索问题，即在最大化累积性能的同时满足安全约束。该挑战在长时程任务中尤为突出，因现有安全方法受限于误差累积和探索能力不足。为此，我们提出一种结合可学习世界模型与双策略（高层与低层）的分层方法，使安全机制贯穿两级。高层策略生成中间子目标，引导探索向安全区域；低层策略利用世界模型进行想象式轨迹推演，减少抵达子目标过程中的不安全行为。在具有高维动作空间的复杂长时程导航与操作任务上评估，本方法显著优于现有安全强化学习基线，在成功率与强约束满足度上表现更优，且在所有随机种子下均严格遵守预设安全预算，而此前方法无法有效解决此类复杂场景。
原文摘要 · Abstract (English)
This work investigates the safe exploration problem in reinforcement learning, where an agent must maximize cumulative performance while simultaneously satisfying safety constraints. This challenge becomes even more pronounced in long-horizon tasks, where existing safe methods face fundamental limitations due to compounding estimation errors and restricted exploration capabilities. To address this problem, we propose a method that combines a learnable world model with two complementary policies a high-level policy and a low-level policy to promote safety at both hierarchical levels. The high-level policy generates intermediate subgoals that bias exploration toward safe regions, while the low-level policy uses imagined rollouts in the learned world model to reduce unsafe behaviors when reaching these subgoals. The proposed method was evaluated on challenging long-horizon navigation and manipulation tasks with high-dimensional action spaces, where it significantly outperforms existing Safe RL baselines in both success rate and strong empirical constraint satisfaction, consistently meeting the prescribed safety budget across seeds, while prior approaches fail to effectively solve these complex long-horizon scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。