提出新方法让强化学习更安全，能动态调整风险约束。
Utility-Constrained Policy Optimization
- 用效用约束框架替代传统约束，支持风险敏感的策略优化。
- 无需提前设定约束上限，训练后可自由调整且无需重训练。
- 在Safety Gymnasium上表现优于或媲美现有基线方法。
约束马尔可夫决策过程（CMDPs）广泛用于将安全性融入强化学习代理，但该框架不支持风险敏感型约束，可能导致最优解混合罕见灾难性行为与频繁过度保守行为。已有实证表明，施加更严格的、风险敏感的约束反而能在风险中立评估下提升性能。效用约束马尔可夫决策过程（UCMDPs）是更合适的框架，但此前缺乏实用解决方案。本文提出一种简单而强大的方法，可在不预先固定约束限值的情况下实现风险敏感约束（只需已知合理范围），从而提升策略灵活性；实践中可零成本调整约束限值。该框架在多个Safety Gymnasium基准任务中表现出色，持续匹配或超越现有基线。
原文摘要 · Abstract (English)
Constrained MDPs (CMDPs) are a widely adopted framework for incorporating safety into RL agents; however, the framework does not support risk-sensitive constraints. This can be problematic: For example, CMDPs allow for optimal solutions that, in order to satisfy the risk-neutral constraints, mix infrequent catastrophic behaviors and frequent, overly conservative ones. Moreover, prior empirical results suggest that enforcing stricter, risk-sensitive constraints can improve performance even under risk-neutral evaluation. The natural framework to incorporate risk-sensitive constraints is utility-constrained MDPs (UCMDPs), but no practical solutions for this problem existed. In this work, we introduce a simple yet powerful methodology for UCMDPs and constrained RL. Besides allowing for risk-sensitive constraints, our framework does not require us to fix constraint limits in advance of training the agent, provided that a sensible range is known. This increases policy flexibility and, in practice, allows for adjustments to these limits at no extra training cost. Besides benefiting from the generality of the framework, our agent shows strong performance in practice, consistently matching or outperforming existing baselines in several Safety Gymnasium benchmark tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。