用不确定性感知的控制方法，让强化学习在探索时更安全且不降低性能。
CAPSULE: Control-Theoretic Action Perturbations for Safe Uncertainty-Aware Reinforcement Learning
- 基于离线学习的概率化控制仿射模型，显式建模系统不确定性。
- 通过控制屏障函数约束动作修正，使安全违规减少但任务表现不下降。
- 适合需要高安全性保障的复杂连续控制场景，如机器人操控。
在动态未知的高维系统中实现安全探索仍是重大挑战。现有安全强化学习方法通常仅提供期望层面的安全保障，仍可能导致安全违规。控制理论方法虽能提供硬性约束保障，但通常依赖已知系统动态或需精确估计控制仿射模型。本文提出一种安全强化学习框架：在离线阶段学习概率化控制仿射动力学模型，并利用该模型显式构建融合模型不确定性的控制屏障函数（CBFs），以生成保守的安全约束。这些约束通过在线动作修正机制强制执行，实现安全探索的同时不过度限制任务性能。在非线性、复杂连续控制基准上的实验表明，本方法在性能上与现有基线相当，同时显著降低安全违规次数。
原文摘要 · Abstract (English)
Ensuring safe exploration in high-dimensional systems with unknown dynamics remains a significant challenge. Existing safe reinforcement learning methods often provide safety guarantees only in expectation, which can still lead to safety violations. Control-theoretic approaches, in contrast, offer hard constraint-based safety guarantees but typically assume access to known system dynamics or require accurate estimation of control-affine models. In this paper, we propose a safe reinforcement learning framework that learns a probabilistic control-affine dynamics model in an offline setting. The learned model is leveraged to explicitly construct control barrier functions (CBFs) that incorporate model uncertainty to provide conservative safety constraints. These CBF constraints are enforced through an online constraint-based action correction mechanism, enabling safe exploration without overly restricting task performance. Empirical evaluations on nonlinear, complex continuous-control benchmarks demonstrate that our approach achieves returns comparable to those of existing baselines while significantly reducing safety violations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。