揭示Transformer训练分两阶段：先逃逸小初始化，再发生秩坍缩。
From Condensation to Rank Collapse: A Two-Stage Analysis of Transformer Training Dynamics
- 用梯度流框架分析线性化Transformer，发现训练分两阶段进行。
- 初始阶段权重扰动维持梯度活跃，后期关键-查询矩阵对齐导致秩坍缩。
- 解释了小初始化为何提升推理能力，适合研究模型训练机制者阅读。
尽管基于Transformer的模型展现出卓越的实证性能，但其训练动态的基本原理仍缺乏超越特定配置研究的系统刻画。受语言模型在小初始化尺度下推理能力提升的实证启发，我们采用[ Zhou et al. NeurIPS 2022 ]建立的梯度流分析框架，系统研究线性化Transformer的训练动态。理论分析将注意力模块的动态分解为两个不同阶段：第一阶段，随机初始化带来的非对称权重扰动维持参数矩阵中非退化的梯度动态，促进从微小初始化区域的系统性逃离；随后，这些矩阵经历凝聚过程，逐步对齐至目标方向。第二阶段，此前静态的关键-查询矩阵主动参与训练，推动归一化矩阵趋向渐近秩坍缩。该两阶段框架推广了经典的定向收敛结果。
原文摘要 · Abstract (English)
Although transformer-based models have shown exceptional empirical performance, the fundamental principles governing their training dynamics are inadequately characterized beyond configuration-specific studies. Inspired by empirical evidence showing improved reasoning capabilities under small initialization scales in language models, we employ the gradient flow analytical framework established in [Zhou et al. NeurIPS 2022] to systematically investigate linearized Transformer training dynamics. Our theoretical analysis dissects the dynamics of attention modules into two distinct stages. In the first stage, asymmetric weight perturbations from random initialization sustain non-degenerate gradient dynamics in parameter matrices, facilitating systematic escape from small initialization regimes. Subsequently, these matrices undergo condensation, progressively aligning toward the target orientation. In the second stage, the previously static key-query matrices actively participate in training, driving the normalized matrices toward asymptotic rank collapse. This two-stage framework generalizes classical directional convergence results.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。