用随机矩阵理论分析注意力矩阵，实现无需验证集的早停
Early-stopping for Transformer model training
- 通过注意力矩阵幂律拟合划分训练三阶段
- 发现浅层注意力谱密度趋于重尾分布
- 提出无验证集依赖的收敛判断指标
本研究基于随机矩阵理论（RMT），提出一种针对Transformer训练动态的新型早停策略。利用注意力矩阵的幂律（PL）拟合作为探测工具，将训练过程划分为三个阶段：结构探索、重尾结构稳定和收敛饱和。实验发现，浅层自注意力矩阵$V$的谱密度始终演化为重尾分布。关键在于，我们提出了两个一致且无需验证集的判据：一个用于量化重尾动态的定量指标，以及一个指示收敛的新型谱特征。这两个判据间的强一致性凸显了RMT在监控和诊断Transformer训练进展中的有效性。
原文摘要 · Abstract (English)
This work, based on Random Matrix Theory (RMT), introduces a novel early-stopping strategy for Transformer training dynamics. Utilizing the Power Law (PL) fit to tansformer attention matrices as a probe, we demarcate training into three stages: structural exploration, heavy-tailed structure stabilization, and convergence saturation. Empirically, we observe that the spectral density of the shallow self-attention matrix $V$ consistently evolves into a heavy-tailed distribution. Crucially, we propose two consistent and validation-set-free criteria: a quantitative metric for heavy-tailed dynamics and a novel spectral signature indicative of convergence. The strong alignment between these criteria highlights the utility of RMT for monitoring and diagnosing the progression of Transformer model training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。