解释为何量化权重时符号翻转比幅度变化影响更小。
A Geometric Analysis of Sign-Magnitude Asymmetry in a ReLU + RMSNorm Block under Ternary Quantization
- 通过几何分解分析符号与幅度扰动的差异。
- 符号翻转产生的输出能量是幅度扰动的2.75倍。
- 适合研究量化模型稳定性与神经网络几何特性的人。
具有RMSNorm的预归一化Transformer在使用{-1,0,+1}三值量化时，损失极小（Ma et al., 2024）。本文通过权重扰动的符号-幅度分解，从几何角度给出解释。在两层ReLU + RMSNorm模型中，当权重服从独立同分布高斯分布且翻转率 $p o 0$ 时，符号翻转产生的横向输出能量是等弗罗贝尼乌斯范数符号保持幅度扰动的 $π/(π-2) imes ≈ 2.75$ 倍（定理3）。机制在于：ReLU在隐空间引入方向不对称性，而RMSNorm的横向投影弗雷歇导数选择性地暴露此差异。符号量化误差本身是符号保持扰动，其角度对齐 $ ext{cos}^2 → 2/π$；其后接ReLU的径向分量（0.365）与前接值 $1-2/π$ 相差仅0.4%，表明ReLU对三值误差近似透明。多层叠加的2.75倍因子未在实验中支持，真实模型的符号敏感度差异源于违反去局域化的异常特征。对于输入维度幅度 $α$，单次符号翻转使后接ReLU的能量放大 $R ≈ nα^2$。在TinyLlama-1.1B上，线性响应范围（$p ≤ 0.5 ext{%}$）内，计数匹配的NLL杠杆稳定在 ∼10× ≈ $n ext{E}[α^2]$，与单条理论一致；全列NLL比值为5.0×，在 $R_{ ext{col}} ≤ 19$ 范围内（67× PPL差距反映度量非线性）。第12层测量的异常 $α$（中位数0.024，最大0.26）证实重尾集中。布斯加恩常数 $2/π$、RMSNorm几何结构与ReLU半空间结构共同解释了预归一化模型中的符号-幅度不对称性，其中 $R ∝ nα^2$ 解释了真实模型的偏差。
原文摘要 · Abstract (English)
Pre-norm Transformers with RMSNorm tolerate ternary {-1,0,+1} weight quantization with surprisingly small loss (Ma et al., 2024). We give a geometric explanation via sign-magnitude decomposition of weight perturbations. In a two-layer ReLU + RMSNorm model with i.i.d. Gaussian weights, sign-flips produce $π/(π-2) \approx 2.75$ times more transverse output energy than sign-preserving magnitude perturbations of equal Frobenius norm, as the flip rate $p \to 0$ (Theorem 3). The mechanism: ReLU creates a hidden-space directional asymmetry between the two perturbation types, which RMSNorm's transverse-projection Fréchet derivative selectively exposes. Sign-quantization error is itself a sign-preserving perturbation with angular alignment $\cos^2 \to 2/π$ (Theorem 4); its post-ReLU radial fraction ($0.365$) matches the pre-ReLU value $1-2/π$ within $0.4\%$, so ReLU is approximately transparent to ternary error. Multi-layer compounding of the $2.75\times$ factor is not experimentally supported; the gap to real-model sign sensitivity arises from outlier features violating delocalization. For an input dimension with amplitude $α$, a single sign-flip produces post-ReLU energy amplified by $R \approx nα^2$ relative to a delocalized entry. On TinyLlama-1.1B, at linear response ($p \leq 0.5\%$), count-matched NLL leverage stabilizes at $\sim 10\times \approx n\mathbb{E}[α^2]$, matching the per-entry theory; the all-column NLL ratio of $5.0\times$ falls within $R_{\mathrm{col}} \leq 19$ ($67\times$ PPL gap reflects metric nonlinearity). Measured outlier $α$ at layer 12 (median $0.024$, max $0.26$) confirms heavy-tailed concentration. The Bussgang constant $2/π$, RMSNorm geometry, and ReLU half-space structure together explain sign-magnitude asymmetry in pre-norm models, with $R \propto nα^2$ accounting for real-model deviations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。