通过权重归一化让CrossQ模型在更高更新率下稳定训练，提升样本效率。
Scaling CrossQ with Weight Normalization
- 在CrossQ中引入权重归一化，解决高更新率下的训练不稳问题。
- 在DeepMind控制基准上，多任务表现优于或媲美原方法，尤其在复杂环境表现突出。
- 无需重置网络等激进操作，适合追求高效稳定的强化学习研究者。
强化学习虽已取得显著进展，但样本效率仍是实际应用的瓶颈。近期的CrossQ方法以极低的更新-数据比（UTD=1）实现了顶尖的样本效率。本文研究了当提高UTD值时CrossQ的可扩展性。我们发现高UTD会加剧训练动态中的问题，特别是Q值偏差爆炸和评价网络权重幅度过大。为此，我们在CrossQ框架中引入权重归一化，该方法能稳定训练过程，防止潜在的可塑性丧失，并保持有效学习率恒定。所提方法在增加的UTD下仍能可靠扩展，在DeepMind控制基准的多个挑战性任务中表现优异，尤其在复杂的狗和人形机器人环境中表现突出。本工作避免了如网络重置等剧烈干预手段，为无模型强化学习提供了提升样本效率与可扩展性的稳健路径。
原文摘要 · Abstract (English)
Reinforcement learning has achieved significant milestones, but sample efficiency remains a bottleneck for real-world applications. Recently, CrossQ has demonstrated state-of-the-art sample efficiency with a low update-to-data (UTD) ratio of 1. In this work, we explore CrossQ's scaling behavior with higher UTD ratios. We identify challenges in the training dynamics which are emphasized by higher UTDs, particularly Q-bias explosion and the growing magnitude of critic network weights. To address this, we integrate weight normalization into the CrossQ framework, a solution that stabilizes training, prevents potential loss of plasticity and keeps the effective learning rate constant. Our proposed approach reliably scales with increasing UTD ratios, achieving competitive or superior performance across a range of challenging tasks on the DeepMind control benchmark, notably the complex dog and humanoid environments. This work eliminates the need for drastic interventions, such as network resets, and offers a robust pathway for improving sample efficiency and scalability in model-free reinforcement learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。