用不确定性估计提升残差强化学习效率，支持随机策略并实现实体部署。
Accelerating Residual Reinforcement Learning with Uncertainty Estimation
- 基于基策略的不确定性聚焦探索，提升样本效率。
- 改进离线残差学习，可感知基策略动作，适配随机策略。
- 在仿真与真实机器人上均实现零样本迁移，性能超越现有方法。
残差强化学习（Residual RL）通过学习轻量级残差策略来修正预训练策略，相比全模型微调更具样本效率。然而，现有方法在稀疏奖励场景下表现不佳，且仅适用于确定性基策略。本文提出两项改进：首先，利用基策略的不确定性估计，引导探索至其置信度低的区域；其次，对离线残差学习进行简单修改，使其能观测基策略的动作，更好地处理随机策略。我们在Robosuite和D4RL的任务上，使用基于高斯和扩散模型的随机基策略进行了评估，并与最先进的微调方法、示范增强强化学习方法及其它残差强化学习方法对比。结果表明，本方法在多种仿真基准环境中显著优于基线。此外，我们在真实世界中部署了学习到的策略，验证了其在零样本模拟到现实迁移中的鲁棒性。
原文摘要 · Abstract (English)
Residual Reinforcement Learning (RL) is a popular approach for adapting pretrained policies by learning a lightweight residual policy that provides corrective actions. While Residual RL is more sample-efficient than finetuning the entire base policy, existing methods struggle with sparse rewards and are designed for deterministic base policies. We propose two improvements to Residual RL that further enhance its sample efficiency and make it suitable for stochastic base policies. First, we leverage uncertainty estimates of the base policy to focus exploration on regions in which the base policy is not confident. Second, we propose a simple modification to off-policy residual learning that allows it to observe base actions and better handle stochastic base policies. We evaluate our method with both Gaussian-based and Diffusion-based stochastic base policies on tasks from Robosuite and D4RL, and compare against state-of-the-art finetuning methods, demo-augmented RL methods, and other residual RL methods. Our algorithm significantly outperforms existing baselines in a variety of simulation benchmark environments. We also deploy our learned polices in the real world to demonstrate their robustness with zero-shot sim-to-real transfer. Paper homepage : lakshitadodeja.github.io/uncertainty-aware-residual-rl/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。