通过球面归一化让强化学习模型更大更稳定，57个任务表现领先。
Hyperspherical Normalization for Scalable Deep Reinforcement Learning
- 用球面约束权重和特征范数，防止训练发散。
- 在4个领域57个连续控制任务上达到新最优性能。
- 适合想放大模型规模的强化学习研究者使用。
增大模型规模和计算量在监督学习中持续提升性能，但在强化学习中常因非平稳数据导致过拟合与优化不稳而失效。为此，我们提出SimbaV2，一种新型强化学习架构：(i) 通过超球面归一化约束权重与特征范数增长；(ii) 采用分布值估计结合奖励缩放，维持不同奖励幅度下的梯度稳定。以软演员-评论家算法为基础，SimbaV2能有效扩展至更大模型与更多计算资源，在4个领域的57个连续控制任务上实现当前最优表现。代码已公开于 https://dojeon-ai.github.io/SimbaV2。
原文摘要 · Abstract (English)
Scaling up the model size and computation has brought consistent performance improvements in supervised learning. However, this lesson often fails to apply to reinforcement learning (RL) because training the model on non-stationary data easily leads to overfitting and unstable optimization. In response, we introduce SimbaV2, a novel RL architecture designed to stabilize optimization by (i) constraining the growth of weight and feature norm by hyperspherical normalization; and (ii) using a distributional value estimation with reward scaling to maintain stable gradients under varying reward magnitudes. Using the soft actor-critic as a base algorithm, SimbaV2 scales up effectively with larger models and greater compute, achieving state-of-the-art performance on 57 continuous control tasks across 4 domains. The code is available at https://dojeon-ai.github.io/SimbaV2.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。