用谱分析揭示大模型训练中的隐藏机制
Spectral Lens: Activation and Gradient Spectra as Diagnostics of LLM Optimization
- 通过激活与梯度谱分析，诊断模型优化过程
- 早期激活谱尾部可预测下游生成效率
- 适用于研究训练动态与模型架构改进
训练损失和吞吐量可能掩盖语言模型内部表征的差异。为揭示这些隐藏机制，我们采用谱测量作为实用且可操作的诊断工具。基于修改版NanoGPT代码库的解码器仅模型家族，我们提出一种基于激活协方差和样本级梯度SVD谱的实证协议。该双视角揭示三项实证发现与一个机制解释：第一，批量大小是表征几何的潜在决定因素——达到相同损失的训练运行会收敛到系统不同的激活谱；第二，训练初期的激活协方差尾部能可靠预测下游词元效率；第三，激活谱头部（主导模式）与梯度谱的移动共同刻画学习动力学变化，可区分学习侧架构改进与执行侧性能提升。这些预测与诊断信号在12、36、48层模型中均持续存在。最后，一个机制模型验证了主要观察结果，并解释了激活协方差谱如何与任务对齐特征学习相关联。
原文摘要 · Abstract (English)
Training loss and throughput can hide distinct internal representation in language-model training. To examine these hidden mechanics, we use spectral measurements as practical and operational diagnostics. Using a controlled family of decoder-only models adapted from the modded NanoGPT codebase, we introduce an empirical protocol based on activation covariance and per-sample gradient SVD spectra. This dual-view reveals three empirical findings and one mechanistic explanation. First, batch size acts as a latent determinant of representation geometry: runs that reach equal loss settle into systematically distinct activation spectra. Second, the activation covariance tail measured early in training reliably forecasts downstream token efficiency. Third, movement of the activation spectrum head (leading modes), together with gradient spectra, characterizes underlying learning-dynamics changes, separating learning-side architectural improvements from primarily execution-side gains. These predictive and diagnostic signals persist across the 12-, 36-, and 48-layer model tiers. Finally, a mechanistic model proves the main observations and explains how activation covariance spectra correlate with task-aligned feature learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。