发现训练轨迹的信号噪声分界，揭示模型优化的内在几何规律。
Spectral Edge Dynamics of Training Trajectories: Signal--Noise Geometry Across Scales
- 用滚动窗口SVD分析参数更新，识别信号与噪声的分界线。
- 三阶段演化模式普遍存在于不同规模模型中，有效信号秩随任务复杂度变化。
- 可提前600-1700步预警模型泛化能力突现，适合研究训练动力学者。
尽管拥有数亿参数，Transformer的训练轨迹仅在少数连贯方向上演化。我们提出谱边动态（SED）来量化这一结构：通过参数更新的滚动窗口SVD，发现一个清晰的谱边——即相干优化方向与随机噪声之间的边界，由最大连续奇异值比 $σ_k / σ_{k+1}$ 确定。在5100万参数的TinyStories模型（4个种子）和1.24亿参数的GPT-2上，谱边呈现普适的三相模式（上升、平台、坍塌）。有效信号秩随任务复杂度自适应调整（5100万参数时 $k^* = 2$，1.24亿参数时 $k^* = 3$），且谱几何与验证损失的方向耦合随窗口大小反转——滞后翻转反映了轨迹积分的时间尺度。通过Johnson–Lindenstrauss投影至 $d = 10W$ 维（如 $W=10$ 时 $d=100$），谱隙保持在5.7%以内，使该框架适用于任意规模模型。在相关工作中，相同的谱几何结构可作为早期预警信号，在模运算、Dyck语言及SCAN基准上，提前600至1700步预测泛化突现。
原文摘要 · Abstract (English)
Despite hundreds of millions of parameters, transformer training trajectories evolve within only a few coherent directions. We introduce Spectral Edge Dynamics (SED) to quantify this structure: a rolling-window SVD of parameter updates reveals a sharp boundary -- the spectral edge -- between coherent optimization directions and stochastic noise, identified via the maximum consecutive singular value ratio $σ_k / σ_{k+1}$. Across a 51M-parameter TinyStories model (4 seeds) and GPT-2 124M under distribution shift, the spectral edge exhibits a universal three-phase pattern (rise, plateau, collapse). The effective signal rank adapts to task complexity ($k^* = 2$ at 51M, $k^* = 3$ at 124M), and the directional coupling between spectral geometry and validation loss reverses with window size -- a lag flip reflecting the timescale of trajectory integration. Johnson--Lindenstrauss projection to $d = 10W$ dimensions (e.g., $d = 100$ for $W = 10$) preserves the spectral gap within $5.7\%$, making the framework applicable to models of arbitrary scale. In companion work, the same spectral geometry provides early-warning signals of grokking -- predicting generalization $600$--$1{,}700$ steps before it occurs across modular arithmetic, Dyck languages, and the SCAN benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。