让离线强化学习在测试时自动变安全，不重训练也能避险。
Lyapunov-Guided Self-Alignment: Test-Time Adaptation for Offline Safe Reinforcement Learning
- 用自对齐机制生成安全轨迹，通过李雅普诺夫条件筛选可行路径
- 在Safety Gymnasium和MuJoCo上成本降低、失败率下降，回报不降反升
- 适合部署后需快速适应的高危场景，如机器人控制、自动驾驶
离线强化学习智能体在实际部署时常因训练数据与真实环境的差距而产生不安全行为。为此，我们提出SAS（安全自对齐），一种基于Transformer的框架，实现无需重训练的测试时适应。SAS的核心是自对齐机制：测试时，预训练智能体生成多条想象轨迹，并选择满足李雅普诺夫条件的可行片段；这些片段被作为上下文提示回传给模型，使智能体在不更新参数的前提下重新校准行为以确保安全。本质上，SAS将李雅普诺夫引导的想象转化为控制不变提示，其Transformer结构支持分层强化学习解释，其中提示机制相当于对潜在技能的贝叶斯推断。在Safety Gymnasium和MuJoCo基准上，SAS持续降低代价与失败率，同时保持或提升回报性能。
原文摘要 · Abstract (English)
Offline reinforcement learning (RL) agents often fail when deployed, as the gap between training datasets and real environments leads to unsafe behavior. To address this, we present SAS (Self-Alignment for Safety), a transformer-based framework that enables test-time adaptation in offline safe RL without retraining. In SAS, the main mechanism is self-alignment: at test time, the pretrained agent generates several imagined trajectories and selects those satisfying the Lyapunov condition. These feasible segments are then recycled as in-context prompts, allowing the agent to realign its behavior toward safety while avoiding parameter updates. In effect, SAS turns Lyapunov-guided imagination into control-invariant prompts, and its transformer architecture admits a hierarchical RL interpretation where prompting functions as Bayesian inference over latent skills. Across Safety Gymnasium and MuJoCo benchmarks, SAS consistently reduces cost and failure while maintaining or improving return.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。