提出新方法在离线强化学习中实现安全约束下的高奖励策略。
Online Optimization for Offline Safe Reinforcement Learning
- 将问题建模为极小极大优化，结合离线与在线算法求解。
- 在严格成本预算下可靠满足安全约束，同时获得高奖励。
- 无需离线策略评估，适配任意离线强化学习算法。
我们研究离线安全强化学习（OSRL）问题，目标是在固定数据上学习最大化奖励的策略，同时满足累积成本约束。本文提出一种新型OSRL方法，将问题表述为极小极大目标，并通过结合离线强化学习与无遗憾在线优化算法求解。当与近似离线强化学习代理和无遗憾在线优化结合时，该方法具有近似最优性。我们还提出一种实用近似方案，可与任意离线强化学习算法配合使用，无需离线策略评估。在DSRL基准上的实验表明，该方法在严格成本预算下能可靠执行安全约束，同时实现高奖励。代码已公开于https://github.com/yassineCh/O3SRL。
原文摘要 · Abstract (English)
We study the problem of Offline Safe Reinforcement Learning (OSRL), where the goal is to learn a reward-maximizing policy from fixed data under a cumulative cost constraint. We propose a novel OSRL approach that frames the problem as a minimax objective and solves it by combining offline RL with online optimization algorithms. We prove the approximate optimality of this approach when integrated with an approximate offline RL oracle and no-regret online optimization. We also present a practical approximation that can be combined with any offline RL algorithm, eliminating the need for offline policy evaluation. Empirical results on the DSRL benchmark demonstrate that our method reliably enforces safety constraints under stringent cost budgets, while achieving high rewards. The code is available at https://github.com/yassineCh/O3SRL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。