位置编码方式决定注意力谱特性,影响模型推理能力。
Fingerprint, Not Blueprint: How Positional Schemes Set the Default Spectral Algebra of Attention

- 通过分析注意力矩阵的谱结构,揭示位置编码如何塑造注意力行为
- 旋转性谱特征仅在RoPE中出现,且与模型功能演化同步而非先验
- 不同位置编码方案导致不同的计算成本,影响模型训练效率
注意力头的预软最大值分数是学习算子M = W_q^T W_k的双线性形式。由于M通常非对称,因此非正规,具有复特征谱和非正交特征向量,适用于非厄米和随机矩阵理论。本文从静态、动态和因果三个层面研究该谱所编码的信息,涵盖七种预训练模型及三种位置编码方案(RoPE、learned-absolute、ALiBi)。静态分析显示,最强的前项词头在RoPE下呈谱旋转态,而在其他方案下为非旋转或内容型;模型层面的区分在所有考察的top-k中均完美实现(精确排列检验p=0.029),且在三个RoPE模型中消除每频段的RoPE虚部可完全破坏诱导能力。动态分析表明,所有头均起始于随机矩阵(Ginibre)零点,旋转特征随行为出现而显现,群体中位抑制发生在电路形成之后,说明最终谱特征是功能凝聚后的指纹而非前置约束。因果实验显示,即使移除谱通道,受限两层训练仍可完整保留能力,但形成延迟显著(四次注册对比,q_BH ≤ 0.016)。成本结构揭示各方案默认差异:强制对称使learned-absolute模型速度下降2.9倍,而尽管静态对称,RoPE头仍可通过相位通道定向路由,绝对位置无法实现。在所考察范围内,位置编码方案决定了注意力头解的默认谱代数:一个由功能后塑的指纹,而非先验硬约束。
原文摘要 · Abstract (English)
The pre-softmax score of an attention head is a bilinear form $score(i,j) = x_i^T M x_j$ in a learned operator $M = W_q^T W_k$. Because M is generally non-symmetric, hence non-normal, it has a complex eigenspectrum and non-orthogonal eigenvectors, the regime where non-Hermitian and random-matrix tools apply. We ask what this spectrum encodes, at three levels for previous-token and induction circuits. Statically, across seven pretrained models spanning three positional schemes, the strongest previous-token heads are spectrally rotational under RoPE and non-rotational, or content-like, where position enters outside QK (learned-absolute and ALiBi); the model-level separation is perfect at every top-k examined (exact permutation $p=0.029$), and zeroing the per-frequency RoPE phase $Im(M_t)$ eliminates induction on a pre-identified previous-token head in all three RoPE models. Dynamically, over public Pythia checkpoints every head originates at the random-matrix (Ginibre) null; the rotational signature emerges with the behavior, not before it, and the population-median suppression that yields the final profile follows circuit formation, so the profile is a consolidated fingerprint, not a precursor. Causally, and at toy scale, no spectral channel is necessary: constrained two-layer training reroutes around every ban with capability intact, albeit at a significant formation delay (four pre-registered contrasts, $q_BH <= 0.016$). The cost structure exposes each scheme's default: imposing symmetry slows learned-absolute models by a factor of 2.9, whereas a RoPE head with a fully symmetric static M still routes directionally via the phase channel, impossible under absolute positions. Within the settings examined, the positional scheme sets the default spectral algebra of an attention head's solution: a fingerprint sculpted after function, not a hard constraint upon it.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。