通过关注策略对参数扰动的敏感度，让强化学习更安全地探索未知区域。
SHAPO: Sharpness-Aware Policy Optimization for Safe Exploration
- 用参数扰动下的梯度评估来感知不确定性，引导保守更新。
- 在多个连续控制任务中，安全性和性能均优于基线方法。
- 适合需高安全性的机器人、自动驾驶等实际部署场景。
安全探索是将强化学习代理应用于安全关键领域的重要前提。本文从认知不确定性视角出发，将智能体对参数扰动的敏感度作为高不确定性区域的实用代理。提出了一种敏锐度感知的策略优化方法（SHAPO），该方法在扰动参数处评估梯度，使策略更新对认知不确定性持悲观态度。理论上证明，这一调整隐式重加权了策略梯度，放大罕见危险动作的影响，同时减弱已知安全动作的贡献，从而在未充分探索区域偏向保守行为。在多个连续控制任务中，本方法显著提升了安全性和任务表现，大幅扩展了现有方法的帕累托前沿。
原文摘要 · Abstract (English)
Safe exploration is a prerequisite for deploying reinforcement learning (RL) agents in safety-critical domains. In this paper, we approach safe exploration through the lens of epistemic uncertainty, where the actor's sensitivity to parameter perturbations serves as a practical proxy for regions of high uncertainty. We propose Sharpness-Aware Policy Optimization (SHAPO), a sharpness-aware policy update rule that evaluates gradients at perturbed parameters, making policy updates pessimistic with respect to the actor's epistemic uncertainty. Analytically we show that this adjustment implicitly reweighs policy gradients, amplifying the influence of rare unsafe actions while tempering contributions from already safe ones, thereby biasing learning toward conservative behavior in under-explored regions. Across several continuous-control tasks, our method consistently improves both safety and task performance over existing baselines, significantly expanding their Pareto frontiers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。