让一个策略零样本适配不同安全阈值，提升离线强化学习安全性与灵活性。
Conditional Sequence Modeling for Safe Reinforcement Learning
- 基于条件序列建模，用单一模型支持多成本阈值的零样本部署。
- 在DSRL基准上显著提升收益-成本权衡，优于主流基线方法。
- 适合需要动态调整安全标准的现实场景部署，如自动驾驶、机器人控制。
离线安全强化学习旨在从固定数据集学习策略，同时在累积成本约束下最大化性能。实际应用中，不同场景对部署要求差异大，需单一策略能零样本适应不同成本阈值。然而，现有大多数离线安全RL方法在预设阈值下训练，导致泛化能力差、部署灵活性不足。受条件序列建模（CSM）进展启发，我们提出RCDT，首个基于CSM的离线安全RL算法，结合拉格朗日式成本惩罚与自适应惩罚系数。为避免过度保守，引入奖励-成本感知轨迹重加权机制和Q值正则化，实现更优的收益-成本权衡。在DSRL基准上的大量实验表明，RCDT持续优于代表性基线，推动了离线安全RL的最新进展。
原文摘要 · Abstract (English)
Offline safe reinforcement learning (RL) aims to learn policies from a fixed dataset while maximizing performance under cumulative cost constraints. In practice, deployment requirements often vary across scenarios, necessitating a single policy that can adapt zero-shot to different cost thresholds. However, most existing offline safe RL methods are trained under a pre-specified threshold, yielding policies with limited generalization and deployment flexibility across cost thresholds. Motivated by recent progress in conditional sequence modeling (CSM), which enables flexible goal-conditioned control by specifying target returns, we propose RCDT, a CSM-based method that supports zero-shot deployment across multiple cost thresholds within a single trained policy. RCDT is the first CSM-based offline safe RL algorithm that integrates a Lagrangian-style cost penalty with an auto-adaptive penalty coefficient. To avoid overly conservative behavior and achieve a more favorable return--cost trade-off, a reward--cost-aware trajectory reweighting mechanism and Q-value regularization are further incorporated. Extensive experiments on the DSRL benchmark demonstrate that RCDT consistently improves return--cost trade-offs over representative baselines, advancing the state-of-the-art in offline safe RL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。