提出新算法减少机器人训练中跌倒,提升安全与效率。
SafeExplorer: An Unbiased Policy Gradient for Reinforcement Learning with Recovery Interventions

- 用改进的PPO在安全区外不依赖重要性采样,消除策略更新偏差。
- 实验显示跌倒次数减少233倍,最终奖励不降反升。
- 适合物理机器人、高风险强化学习任务的研究者使用。
在真实机器人上训练强化学习智能体时,每次跌倒都可能造成硬件损坏且无法重置,因此目标应是尽量减少跌倒,而非像约束马尔可夫决策过程那样权衡跌倒与回报。传统方法在状态偏离安全区域时交由独立恢复策略接管,但由此产生的混合策略轨迹会隐式引入偏差,而重要性采样修正在恢复策略确定时无法定义。本文提出对近端策略优化(PPO)的即插即用改进,其核心是仅在安全时间步使用得分函数的无偏策略梯度估计器,完全不评估恢复策略密度,因此在恢复策略确定时仍有效,而此时重要性采样失效;实证表明,即使恢复策略为随机,该方法也优于重要性采样。此外,通过引入确定性动力学下恢复触发状态的闭式价值和仅在恢复成功时模仿恢复动作的损失函数,进一步加速学习。在三个环境、五组种子的基准测试中,该算法在HalfCheetah、Ant和Unitree Go1上分别将训练期跌倒减少233倍、48倍和26倍,同时达到或超过标准PPO的最终奖励;在恢复策略不可靠的Ant环境中,它是唯一能达成最佳奖励80%的方法。
原文摘要 · Abstract (English)
Training reinforcement-learning agents directly on physical robots makes every fall costly, since a fall can damage the platform and cannot be undone like a simulator reset; the goal is therefore to minimize falls during training rather than trade them off against return, as constrained Markov decision process (MDP) formulations do. A standard mitigation hands control to a separate recovery policy whenever the agent leaves a designer-specified safe region (a subset of state space it should stay within), but the resulting mixed-policy rollouts silently bias every on-policy update, and the importance-sampling correction that would remove this bias is ill-defined whenever the recovery policy is deterministic. We address this bias with a drop-in modification of proximal policy optimization (PPO). Its core is an unbiased policy-gradient estimator that uses the score function only at safe timesteps and never evaluates the recovery policy's density, so it stays valid even when the recovery policy is deterministic, exactly where importance sampling breaks, and it empirically dominates importance sampling even when the recovery policy is stochastic. Because the recovery policy still makes credit assignment slow near the safe-region boundary, two further components accelerate learning: a closed-form value for recovery-triggering states when dynamics and recovery are deterministic, and an imitation loss that copies recovery actions only when recovery succeeds. On a three-environment, five-seed benchmark, the resulting algorithm reduces training-time falls by factors of 233x, 48x, and 26x on HalfCheetah, Ant, and Unitree Go1 over standard PPO, while matching or exceeding PPO's final reward, and on Ant, where the recovery policy is unreliable, it is the only method that reaches 80% of the best final reward.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。