用流模型设计安全策略，离线训练更快、部署更实时。
Safe Flow Q-Learning: Offline Safe Reinforcement Learning with Reachability-Based Flow Policies
- 结合可达性理论构建安全价值函数，通过自洽贝尔曼方程学习。
- 训练流策略并蒸馏为单步动作器，部署无需拒绝采样，延迟极低。
- 在船导航和安全物理仿真任务中，安全约束违反显著减少。
离线安全强化学习旨在从静态数据集中学习最大化奖励的策略，同时满足严格的安全约束。现有方法常依赖软期望成本目标或迭代生成推断，难以满足实时安全控制需求。本文提出安全流Q学习（Safe Flow Q-Learning, SafeFQL），将流策略（FQL）扩展至离线安全RL：利用基于哈密顿-雅可比可达性的安全价值函数，结合高效的单步流策略。SafeFQL通过自洽贝尔曼递归学习安全价值，采用行为克隆训练流策略，并将其蒸馏为单步动作器，实现无拒绝采样的奖励最大化安全动作选择。实验表明，SafeFQL虽离线训练成本略高，但推理延迟远低于扩散类生成基线，适合实时安全关键场景。在船导航与Safety Gymnasium MuJoCo任务中，其性能达到或超越已有离线安全RL方法，同时显著降低约束违反次数。
原文摘要 · Abstract (English)
Offline safe reinforcement learning (RL) seeks reward-maximizing policies from static datasets under strict safety constraints. Existing methods often rely on soft expected-cost objectives or iterative generative inference, which can be insufficient for safety-critical real-time control. We propose Safe Flow Q-Learning (SafeFQL), which extends FQL to safe offline RL by combining a Hamilton--Jacobi reachability-inspired safety value function with an efficient one-step flow policy. SafeFQL learns the safety value via a self-consistency Bellman recursion, trains a flow policy by behavioral cloning, and distills it into a one-step actor for reward-maximizing safe action selection without rejection sampling at deployment. Empirically, SafeFQL trades modestly higher offline training cost for substantially lower inference latency than diffusion-style safe generative baselines, which is advantageous for real-time safety-critical deployment. Across boat navigation, and Safety Gymnasium MuJoCo tasks, SafeFQL matches or exceeds prior offline safe RL performance while substantially reducing constraint violations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。