通过自适应缩放动作，让智能体在探索中更安全且性能更强。
Constraint-Aware Reinforcement Learning via Adaptive Action Scaling
- 用可学习的调节器动态缩放动作，预测并缓解约束违反。
- 在稀疏奖励任务中，约束违规减少126倍，回报提升超10倍。
- 无需外部安全过滤器，兼容SAC、TD3等主流强化学习方法。
安全强化学习旨在通过减少训练期间探索引发的不安全行为来降低约束违规。现有方法通常依赖单一策略联合优化奖励与安全性，易因目标冲突导致不稳定；或使用需先验系统知识的外部安全过滤器。本文提出一种模块化的成本感知调节器，根据预测的约束违规情况对智能体动作进行缩放，通过平滑的动作调制保留探索能力，而非直接覆盖策略。该调节器在最小化约束违规的同时避免动作过度抑制。方法可无缝集成至SAC、TD3等离线策略强化学习算法，在Safety Gym行走任务中实现最优的回报-成本比，相比以往方法，约束违规减少达126倍，回报提升超过一个数量级。
原文摘要 · Abstract (English)
Safe reinforcement learning (RL) seeks to mitigate unsafe behaviors that arise from exploration during training by reducing constraint violations while maintaining task performance. Existing approaches typically rely on a single policy to jointly optimize reward and safety, which can cause instability due to conflicting objectives, or they use external safety filters that override actions and require prior system knowledge. In this paper, we propose a modular cost-aware regulator that scales the agent's actions based on predicted constraint violations, preserving exploration through smooth action modulation rather than overriding the policy. The regulator is trained to minimize constraint violations while avoiding degenerate suppression of actions. Our approach integrates seamlessly with off-policy RL methods such as SAC and TD3, and achieves state-of-the-art return-to-cost ratios on Safety Gym locomotion tasks with sparse costs, reducing constraint violations by up to 126 times while increasing returns by over an order of magnitude compared to prior methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。