让离线强化学习自动切换策略，安全运行更灵活。
Constraint-Adaptive Policy Switching for Offline Safe Reinforcement Learning
- 训练时学多个策略，共享表示并权衡奖励与成本
- 部署时按当前安全约束选最优策略，38项任务表现更优
- 无需重训练，适合需动态调整安全边界的场景
离线安全强化学习（OSRL）旨在从固定数据集学习决策策略，在满足预设安全约束的前提下最大化奖励。然而，在不重新训练的情况下适应部署阶段变化的安全约束仍是未充分探索的挑战。为此，本文提出约束自适应策略切换（CAPS），作为现有离线强化学习算法的封装框架。训练阶段，CAPS利用离线数据学习多个共享表示的策略，分别优化不同的奖励-成本权衡；测试阶段，根据当前成本约束，在各策略中选择未来奖励最高的策略进行执行。在DSRL基准的38个任务上，CAPS持续优于现有方法，建立起一种强有力的基于包装器的OSRL基线。代码已公开于https://github.com/yassineCh/CAPS。
原文摘要 · Abstract (English)
Offline safe reinforcement learning (OSRL) involves learning a decision-making policy to maximize rewards from a fixed batch of training data to satisfy pre-defined safety constraints. However, adapting to varying safety constraints during deployment without retraining remains an under-explored challenge. To address this challenge, we introduce constraint-adaptive policy switching (CAPS), a wrapper framework around existing offline RL algorithms. During training, CAPS uses offline data to learn multiple policies with a shared representation that optimize different reward and cost trade-offs. During testing, CAPS switches between those policies by selecting at each state the policy that maximizes future rewards among those that satisfy the current cost constraint. Our experiments on 38 tasks from the DSRL benchmark demonstrate that CAPS consistently outperforms existing methods, establishing a strong wrapper-based baseline for OSRL. The code is publicly available at https://github.com/yassineCh/CAPS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。