用不确定性指导模仿，让智能体更聪明地学示范。
Uncertainty-Based Smooth Policy Regularisation for Reinforcement Learning with Few Demonstrations
- 通过集成方法建模动作不确定性，动态判断何时模仿示范
- 在8个机器人任务中提升性能，复杂任务最高提升14倍
- 适合示范少、质量差的强化学习场景
在稀疏奖励的强化学习中，示范可加速学习，但何时模仿仍难判断。本文提出基于示范的平滑策略正则化（SPReD），核心问题是：何时应模仿示范，何时应遵循自身策略？SPReD利用集成方法显式建模示范与策略动作的Q值分布，量化不确定性以进行比较。我们设计两种互补的不确定性感知方法：概率方法估计示范更优的可能性，优势方法按统计显著性缩放模仿强度。与现有方法（如Q-filter）的二元模仿决策不同，SPReD采用连续、与不确定性成比例的正则化权重，降低训练过程中的梯度方差。尽管计算简单，实验显示其在8个机器人任务中表现卓越，复杂任务性能最高提升14倍，且对示范质量和数量均具鲁棒性。代码已开源：https://github.com/YujieZhu7/SPReD。
原文摘要 · Abstract (English)
In reinforcement learning with sparse rewards, demonstrations can accelerate learning, but determining when to imitate them remains challenging. We propose Smooth Policy Regularisation from Demonstrations (SPReD), a framework that addresses the fundamental question: when should an agent imitate a demonstration versus follow its own policy? SPReD uses ensemble methods to explicitly model Q-value distributions for both demonstration and policy actions, quantifying uncertainty for comparisons. We develop two complementary uncertainty-aware methods: a probabilistic approach estimating the likelihood of demonstration superiority, and an advantage-based approach scaling imitation by statistical significance. Unlike prevailing methods (e.g. Q-filter) that make binary imitation decisions, SPReD applies continuous, uncertainty-proportional regularisation weights, reducing gradient variance during training. Despite its computational simplicity, SPReD achieves remarkable gains in experiments across eight robotics tasks, outperforming existing approaches by up to a factor of 14 in complex tasks while maintaining robustness to demonstration quality and quantity. Our code is available at https://github.com/YujieZhu7/SPReD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。