提出新型球面归一化方法,提升深层网络训练稳定性。
Sphere Retraction Normalizations

- 将残差连接统一到球面几何框架,通过角度重投影实现归一化
- 新方法在nanoGPT上验证,最优性能出现在有限参数p处而非指数映射
- 仅需代数运算即可保持范数,适合轻量化深度模型应用
残差连接是训练深层神经网络稳定性的标准机制。地测归一化(GeoNorm)将其重构于黎曼流形上,使每一层输出相对于当前隐藏状态正交,并通过黎曼指数映射施加更新。每个隐藏状态保持恒定的ℓ₂-范数,使残差流被限制在超球面上。然而,指数映射只是广义重投影映射家族中的一个成员。我们发现,在超球面上,整个家族退化为单一标量设计选择。不同重投影之间的区别仅在于如何将更新幅度转换为由隐藏状态与更新张成平面内的旋转角。这一视角将欧氏残差连接与GeoNorm统一于同一框架。采用度量投影重投影和Cayley重投影实例化后,得到Proj-SpheretNorm与Cay-SpheretNorm,二者均为范数保持且仅需代数运算。两者均属于单参数角度重投影族p-SpheretNorm,其旋转角趋于饱和而非无限增长。当p=1和p=2时分别精确恢复前述两种方法,而恒等映射与GeoNorm仅为两端极限情况。在nanoGPT上的实验表明,三者均优于现有轻量级深度连接方案,最佳验证损失出现在有限的p值,表明指数映射并非球面残差流的最优重投影,而仅是谱系的一端。
原文摘要 · Abstract (English)
Residual connections are the de facto mechanism for training deep neural networks stably. Geodesic Normalization (GeoNorm) recasts them on a Riemannian manifold, orthogonalizing each layer output against the current hidden state and applying the resulting update through the Riemannian exponential map. Every hidden state thus keeps a constant $\ell_{2}$-norm, confining the residual stream to a hypersphere. The exponential map, however, is only one member of a broad family of retraction maps. We show that on the hypersphere this entire family collapses to a single scalar design choice. What distinguishes one retraction from another is only how the magnitude of an update is converted into a rotation angle within the plane spanned by the hidden state and the update. This view places Euclidean residual connections and GeoNorm in one framework. Instantiating it with the metric projection retraction and the Cayley retraction yields Proj-SpheretNorm and Cay-SpheretNorm, which are exactly norm-preserving yet require only algebraic operations. Both prove to be members of a one-parameter family of angular retractions, $p$-SpheretNorm, whose rotation angle saturates rather than growing without bound. The two methods above are recovered exactly at $p = 1$ and $p = 2$, while the identity map and GeoNorm arise only as limits at either end. On nanoGPT, all three methods outperform existing lightweight deep connection schemes, and the best validation loss is attained at finite $p$, indicating that the exponential map is not the preferred retraction for spherical residual streams but merely one end of a spectrum.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。