arXiv:2607.17822cs.LGcs.AI2026-07

提出新型归一化方法MRSNorm,提升模型稳定性与参数效率。

Phasor Attention: Mean Root Square Normalization for Phase Manifold Preservation

  • 将通道配对为二维相量,先算局部平方根再全局平均,反转传统归一化逻辑。
  • 在CIFAR-100上参数减半仍保持稳定,极端超参下避免梯度爆炸。
  • 天然具备三角梯度裁剪机制,适合追求鲁棒性与低参数量的模型设计。

尽管均方根归一化已成为现代序列模型加速的标准,但其依赖独立标量的平方累加(∑x²)会引发异常值导致的数值不稳定性、梯度饥饿和各向异性相位失真。本文提出均值平方根归一化(MRSNorm),通过将通道结构化配对为二维相量,数学上反转传统缩放范式:先计算局部L₂模长(平方根),再通过全局L₁平均(均值)聚合。该操作严格约束激活值位于相量流形上,保持共形不变性。通过在相量分量间共享单一仿射权重,MRSNorm使可学习参数数减半,证明标准归一化中无约束的空间缩放是一种有害冗余。我们从理论上证明,这种几何约束带来内置的三角梯度裁剪器,受毕达哥拉斯恒等式控制,无条件等化局部梯度范数,实现梯度同质性。在带有残差网络的CIFAR-100上的实证表明,尽管参数减半,MRSNorm在严苛压力测试中仍提供关键结构稳定性。在标准归一化出现梯度发散的极端超参数设置下,MRSNorm成功防止数值爆炸,保障稳定优化轨迹。研究结果提出一种基于相量的深度表征学习根本范式转变。MRSNorm实现详见附录C。

原文摘要 · Abstract (English)

While Root Mean Square Normalization has become the de facto standard for accelerating modern sequence models, its reliance on the quadratic accumulation of independent scalars ($\sum x^2$) inherently triggers outlier-induced numerical instability, gradient starvation, and anisotropic phase distortion. We introduce Mean Root Square Normalization (MRSNorm). By structurally pairing channels into 2D phasors, MRSNorm mathematically inverts the traditional scaling paradigm: it computes the localized $L_2$ magnitudes (Root Square) before aggregating them via a global $L_1$ average (Mean). This operational inversion strictly constrains activations to a phasor manifold, preserving conformal invariance. By sharing a single affine weight across phasor components, MRSNorm halves the total number of learnable parameters, proving that unconstrained spatial scaling in standard norms is a harmful redundancy. We analytically demonstrate that this geometric constraint yields a built-in, trigonometric gradient clipper governed by the Pythagorean identity, unconditionally equalizing the local gradient norm to ensure Gradient Homogeneity. Empirical evaluations on a ResNet with CIFAR-100 show that despite halved parameters, MRSNorm provides critical structural stability under rigorous stress tests. Under extreme hyperparameter settings where standard normalizations suffer from gradient divergence, MRSNorm successfully prevents numerical explosion and secures stable optimization trajectories. Our findings propose a fundamental paradigm shift toward phasor-based deep representation learning. The implementation of MRSNorm is available at Appendix C.

归一化模型稳定相量表示

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。