arXiv:2606.19491cs.LGstat.ML2026-06被引 2

发现层归一化模型中可直接从参数读出的死方向,无需前向计算。

Algebraic Dead Directions in LayerNorm Transformers: A Forward-Pass-Only Diagnostic at LLM Scale

  • 仅通过层归一化缩放参数γ^{-1}即可算出死方向,不需前向或反向传播。
  • 在9个层归一化模型上,初始权重预测与实测死方向一致至小数点后4位。
  • 该方法可快速判断模型归一化类型,适合研究模型结构与优化的学者。

预训练Transformer模型处于损失函数的奇异极小值附近,其参数空间中存在“死方向”——即方向上的方向性费舍尔信息为零。传统方法需前向传播和激活矩阵的特征分解,无法仅由参数得到。本文提出一种仅依赖层归一化缩放参数γ^{-1}/||γ^{-1}||的代数解法,该方向精确对应最终归一化后激活协方差的核空间,构成参数空间中的死方向。无需前向或反向传播,也无需特征值求解,是目前已知最廉价的死方向检测方法。我们在14个预训练模型(9个层归一化、5个RMS归一化,规模160M-35B,涵盖语言与视觉任务)上验证:随机初始化时,该方向与实测最小奇异方向匹配至小数点后四位(9/9),而所有RMS归一化模型(5/5)均未出现此方向,因其缺乏均值减法投影机制。训练后,该方向对应的协方差特征值增强约10^3倍,更多死方向开启;从随机初始化到训练的差异可通过一次前向传播读出该坐标处的奇异结构。由此推导出两个闭式结论:13/14个模型的残差流最小奇异值在块间保持不变,唯一例外(Gemma4-31B)正是该方向所揭示的真实死方向;且该方向的存在可仅凭参数判断模型是否采用层归一化。

原文摘要 · Abstract (English)

Pretrained transformers sit near singular minima of the loss, where the Fisher information metric degenerates along dead directions: directions in parameter space along which the directional Fisher vanishes. Locating such a direction normally needs a forward pass and an eigendecomposition of activations, or a sampling-based complexity estimate; none returns a direction computable from the network's parameters alone. We give one, for LayerNorm transformers. The inverse-scale direction $γ^{-1}/\|γ^{-1}\|$ of the LayerNorm affine is an exact algebraic kernel of the post-final-norm centred activation covariance, for any input distribution, and induces a corresponding dead direction in parameter space. It is read from the LN scale parameter alone, with no forward or backward pass and no eigensolve: the cheapest dead-direction read, specific to LayerNorm. We test it on $14$ pretrained transformers ($9$ LayerNorm, $5$ RMSNorm; $160$M-$35$B; language and vision objectives). At random initialisation the predicted direction matches the measured bottom singular direction (one forward pass, direct SVD) to four decimal places on $9/9$ LayerNorm models, and is correctly absent on $5/5$ RMSNorm models, which lack the mean-subtraction projector that creates it. On the trained checkpoint the covariance eigenvalue along this direction deepens by ${\sim}10^3\times$ and further dead directions open; the random-init-to-trained gap is a one-forward-pass, per-checkpoint readout of singular structure along the predicted coordinate. Two consequences follow in closed form: the residual stream's smallest singular value is preserved block-to-block on $13/14$ transformers measured on their own input distribution, the one exception (Gemma$4$-$31$B) a genuine dead direction the same read pinpoints; and the kernel direction's presence classifies a transformer's normalisation from the parameters alone.

Transformer死方向层归一化结构分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。