让强化学习快速适应新安全约束，无需重训整个模型。
Constrained Reinforcement Learning Using Successor Representations
- 用可学习的权重矩阵解耦动态、奖励与成本，实现灵活更新。
- 环境成本变化时，仅需微调少量参数，训练速度提升显著。
- 适合需要频繁调整安全规则的复杂现实场景，如自动驾驶导航。
现实世界中的强化学习依赖于将安全约束纳入策略。通常通过在马尔可夫决策过程引入额外的成本信号来建模，独立于奖励信号提示代理避免不良行为。然而，现有方法难以适应成本函数的变化，例如领域迁移或障碍物移动。这导致策略过于僵化，无法应对复杂现实条件。本文提出安全深度后继表示（SafeDSR），一种新型约束强化学习方法，可快速重训练以适应新的成本结构。SafeDSR 在深度后继表示（Kulkarni et al., 2016）基础上扩展，引入单一可学习权重矩阵，解耦价值函数在动态、奖励和成本之间的关联。该矩阵可通过监督方式更新，无需重训整个网络。我们在一个自由配置的二维导航环境中验证了其能力，结果表明：在简单导航任务上表现相当，但面对成本结构变化时灵活性显著提升。
原文摘要 · Abstract (English)
Real-world Reinforcement Learning depends on the ability to formulate safety constraints into a policy. A common way to model such constraints is to introduce an additional cost signal in the Markov Decision Process, which notifies the agent of unwanted behavior independently of the reward signal. Unfortunately, current methods are hard to adapt to changes in the cost function introduced by, e.g., domain shift or obstacles moving over time. The lack of adaptability means that policies are too unflexible to deal with complex real-world conditions. We propose the Safe Deep Successor Representation (SafeDSR), a novel method that allows quick retraining of policies towards new cost structures. SafeDSR extends the Deep Successor Representation (Kulkarni et al., 2016) to Constrained Reinforcement Learning by introducing a single learnable weight matrix to decouple the learned value function across dynamics, rewards, and costs. This matrix can be updated in a supervised manner instead of having to adapt the whole network if the cost structure of the environment changes. We demonstrate this ability in a freely configurable two-dimensional navigation environment and show that our method is competitive on a simple navigation task while being considerably more flexible
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。