arXiv:2605.16600cs.LGcs.AI2026-05被引 1

预训练与对齐在Transformer权重空间中留下不同痕迹,揭示了写入与读取路径的几何差异。

Where Pretraining writes and Alignment reads: the asymmetry of Transformer weight space

论文配图:Where Pretraining writes and Alignment reads: the asymmetry of Transformer weight space
图 1 · 摘自论文原文
  • 通过子空间分数探测器分析权重更新方向,发现对齐更新集中在注意力读取路径。
  • 预训练使写入路径具备预测空间方向性,而对齐过程对此影响甚微。
  • 适用于理解大模型内部机制、优化训练策略的研究者。

交叉熵预训练与偏好对齐更新相同的Transformer权重,但在几何上留下截然不同的痕迹。我们通过相对子空间分数探针,追踪权重增量与残差流激活子空间及由未嵌入矩阵定义的预测子空间的对齐情况。对齐增量集中于读取路径($W_Q$, $W_K$),沿注意力输入激活的主要方向;而写入路径($W_O$, $W_2$)相对于预测子空间则近乎各向同性。这一现象由各向异性梯度累积解释:矩阵 $W$ 的更新是外积 $δ_t a_t^ op$ 的和,其方向结构继承自协方差集中的那一侧。对于读取路径,输入激活 $a_t$ 的协方差在训练后呈尖峰状,导致无任务依赖的方向集中;对于写入路径,相关侧是上游梯度 $δ_t$,其各向异性取决于损失函数。交叉熵提供标准的尖锐样本级信号,使预训练期间写入路径形成预测几何;而对齐目标通常无法进一步增强写入侧的集中性。我们通过检查点内轨迹、渐进对比目标控制实验以及方向匹配的闭式秩-1干预,提供了因果证据支持所提出的权重空间几何结构。

原文摘要 · Abstract (English)

Cross-entropy pretraining and preference alignment update the same transformer weights, but leave geometrically distinct traces. We characterise this asymmetry with a relative-subspace-fraction probe that tracks how weight deltas align with residual-stream activation subspaces and with the prediction subspace defined by the unembedding. Alignment deltas concentrate in the read pathway ($W_Q$, $W_K$), along principal directions of attention-input activations, while remaining near-isotropic in the write pathway ($W_O$, $W_2$) relative to the prediction subspace. We explain this pattern through anisotropic gradient accumulation: updates to a matrix $W$ are sums of outer products $δ_t a_t^\top$, and inherit directional structure from whichever side has concentrated covariance. For read-pathway matrices, this side is the input activation $a_t$, whose covariance is spiked in trained transformers and therefore produces objective-agnostic concentration. For write-pathway matrices, the relevant side is the upstream gradient $δ_t$, whose anisotropy depends on the loss. Cross-entropy supplies the canonical sharp per-sample signal, inducing write-pathway prediction geometry during pretraining; alignment objectives typically add little further write-side concentration. We support this explanation with a within-checkpoint trajectory, a graded contrastive-objective control, and a closed-form rank-1 intervention with matched direction controls, providing causal evidence for the proposed weight-space geometry.

Transformer权重空间预训练对齐机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。