揭示Transformer注意力敏感性的几何本质，给出精确的稳定性分析。
Exact Attention Sensitivity and the Geometry of Transformer Stability
- 基于归一化对齐的几何分析，推导出注意力雅可比的精确公式。
- 发现温度与注意力分布均匀性决定敏感度，且在长序列中保持稳定。
- 适用于研究模型训练初期稳定性，尤其关注预归一化结构的差异。
我们针对基于标记粒度计算的Transformer注意力发展了一种敏感性分析。核心结果是：对于温度化的softmax函数 $u\mapsto\mathrm{softmax}(u/τ)$，其雅可比矩阵 $J_τ(u)$ 满足精确恒等式 $\|J_τ(u)\|_{\infty\to1}=θ(p)/τ$，其中 $θ(p)=4\max_{S\subseteq[L]}p(S)(1-p(S))$ 反映注意力分布被均分的能力而非集中程度。结合块-$\infty$/RMS范数，使得行随机注意力混合非扩张。由此得到多头注意力的分布感知局部雅可比界，以及在有界输入集上与序列长度无关的利普希茨界，明确依赖于宽度、输入幅值、温度和投影范数。还识别出归一化位置的结构性差异：预归一化残差子层的雅可比含加法单位项，而后归一化则无。一个层归一化投影引理给出了后归一化展开中仅层归一化项几何收缩的充分条件，但未在实验中验证。在三个774M参数模型的预归一化早期训练运行中，注意力变得显著集中，而每个采样层和检查点的 $θ(p)$ 中位下界始终接近1，表明至少一半采样行具有近最大精确敏感性。少数行进入主导原子态，敏感性较低，符合 $p_{\max}$ 与 $θ(p)$ 的确定关系。
原文摘要 · Abstract (English)
We develop a sensitivity analysis for transformer attention in a geometry aligned with tokenwise computation. Our main result is the exact identity $\|J_τ(u)\|_{\infty\to1}=θ(p)/τ$ for the Jacobian $J_τ(u)$ of the tempered softmax $u\mapsto\mathrm{softmax}(u/τ)$, where $θ(p)=4\max_{S\subseteq[L]}p(S)(1-p(S))$ measures how evenly the attention distribution can be bisected rather than how concentrated it is. We combine this identity with a block-$\infty$/RMS norm under which row-stochastic attention mixing is nonexpansive. This yields a distribution-aware local Jacobian bound for multi-head attention and a sequence-length-independent Lipschitz bound on bounded input sets, with explicit dependence on width, input magnitude, temperature, and projection norms. We also identify a structural distinction between normalization placements: a pre-LN residual-sublayer Jacobian contains an additive identity term, whereas a post-LN residual-sublayer Jacobian does not. A LayerNorm projection lemma gives a sufficient condition under which the LayerNorm-only term in the post-LN expansion contracts geometrically; the condition is not tested by our experiments. Across three Pre-LN early-training runs of $774$M-parameter models, attention becomes substantially more concentrated while the median lower-bound certificate for $θ(p)$ remains near one at every sampled layer and checkpoint. This certifies near-maximal exact sensitivity for at least half of the sampled rows within each layer. A minority of rows enters a dominant-atom regime with lower exact sensitivity, consistent with the deterministic relationship between $p_{\max}$ and $θ(p)$.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。