首次揭示Transformer训练中权重谱的动态演化规律，发现压缩波与谱梯度等新现象。
The Spectral Lifecycle of Transformer Training: Transient Compression Waves, Persistent Spectral Gradients, and the Q/K--V Asymmetry
- 通过每25步追踪全量权重矩阵的奇异值分解，揭示训练过程中的谱动态变化。
- 发现深层模型存在非单调倒U型谱指数梯度，且后期层压缩程度超过早期层。
- 提出谱结构可指导模型剪枝，性能优于传统方法，验证了谱特征的因果作用。
我们首次系统研究了Transformer预训练过程中权重矩阵奇异值谱的变化，以每25步为间隔对三个规模（30M–285M参数）的模型进行完整的奇异值分解。发现了三种现象：(1) 瞬态压缩波：稳定秩压缩从浅层向深层传播，形成先升后降的梯度，晚期层最终压缩程度超过早期层；(2) 持久谱梯度：幂律指数α在深度方向上形成非单调倒U型分布，随层数增加峰值向浅层偏移；(3) Q/K–V功能不对称性：输出投影均匀压缩，而查询/键投影承载全部深度依赖动态。这一解耦表明秩与谱形状编码训练中不同信息。我们构建两时间尺度动力学模型，推导出缩放律（Δα ∝ L^{0.26}，R²=0.99）。在九个模型（30M–1B参数，8–36层）上验证，α能有效预测层重要性（ρ=0.69–0.84，p<0.02），且基于谱引导的剪枝在七个模型中优于末尾剪枝策略1.1倍至3.6倍，最差与最优差距达23.7倍，证实谱结构的因果作用。
原文摘要 · Abstract (English)
We present the first systematic study of weight matrix singular value spectra \emph{during} transformer pretraining, tracking full SVD decompositions of every weight matrix at 25-step intervals across three model scales (30M--285M parameters). We discover three phenomena: \textbf{(1)~Transient Compression Waves:} stable rank compression propagates as a traveling wave from early to late layers, creating a dramatic gradient that peaks early then \emph{reverses} -- late layers eventually over-compress past early layers. \textbf{(2)~Persistent Spectral Gradients:} the power-law exponent~$α$ develops a permanent depth gradient forming a non-monotonic inverted-U in deeper models, with peaks shifting toward earlier layers as depth increases. \textbf{(3)~Q/K--V Functional Asymmetry:} value/output projections compress uniformly while query/key projections carry the full depth-dependent dynamics. The dissociation between transient compression and persistent spectral shape reveals that \emph{rank and spectral shape encode fundamentally different information about training}. We formalize this as a two-timescale dynamical model and derive scaling laws ($Δα\propto L^{0.26}$, $R^2{=}0.99$). We validate on nine models across three families (custom, GPT-2, Pythia; 30M--1B parameters; 8--36 layers), demonstrate that $α$ predicts layer importance ($ρ{=}0.69$--$0.84$, $p{<}0.02$), and show that spectral-guided pruning outperforms Last-N heuristics by $1.1{\times}$--$3.6{\times}$ across seven models in two families (GPT-2 124M--774M, Pythia 160M--1B), with worst-vs-best gaps up to $23.7{\times}$ confirming the causal role of spectral structure.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。