解决残差强化学习中价值函数学习效率低的问题。
What Makes Value Learning Efficient in Residual Reinforcement Learning?
- 用基础策略的转移数据作为隐式预热锚点，缓解冷启动问题。
- 通过批评者归一化恢复对价值差异的敏感度，提升学习效率。
- 方法简单有效，适用于多种策略架构与观测模态。
残差强化学习通过冻结基础策略并仅学习有限修正，实现表达性预训练策略的稳定在线优化。然而，残差强化学习中的价值学习存在独特挑战，尚不明确。本文识别出两个关键瓶颈：冷启动病理，即评论家缺乏对基础策略附近价值景观的知识；结构尺度失配，即残差贡献远小于基础动作。通过系统研究，我们揭示了这些瓶颈的机制，发现简单而合理的解决方案即可：基础策略的转移数据可作为隐式预热的必要锚点，批评者归一化能有效恢复表示敏感性以区分价值差异。基于此，我们提出DAWN（Data-Anchored Warmup and Normalization），一种针对残差强化学习高效价值学习的最小化方法。通过解决上述瓶颈，DAWN在多种基准、策略架构和观测模态下均展现出显著效率提升。
原文摘要 · Abstract (English)
Residual reinforcement learning (RL) enables stable online refinement of expressive pretrained policies by freezing the base and learning only bounded corrections. However, value learning in residual RL poses unique challenges that remain poorly understood. In this work, we identify two key bottlenecks: cold start pathology, where the critic lacks knowledge of the value landscape around the base policy, and structural scale mismatch, where the residual contribution is dwarfed by the base action. Through systematic investigation, we uncover the mechanisms underlying these bottlenecks, revealing that simple yet principled solutions suffice: base-policy transitions serve as an essential value anchor for implicit warmup, and critic normalization effectively restores representation sensitivity for discerning value differences. Based on these insights, we propose DAWN (Data-Anchored Warmup and Normalization), a minimal approach targeting efficient value learning in residual RL. By addressing these bottlenecks, DAWN demonstrates substantial efficiency gains across diverse benchmarks, policy architectures, and observation modalities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。