揭示梯度下降中早期停止的瞬态现象源于输入协方差的各向异性。
Random Matrix Theory of Early-Stopped Gradient Flow: A Transient BBP Scenario
- 基于随机矩阵理论构建线性师生模型，模拟梯度流学习过程。
- 信号峰值在特定时间窗口内显现后又消失，呈现瞬态BBP相变。
- 适用于理解早期停止为何有效，适合机器学习理论研究者。
训练模型的实证研究常发现，在过拟合主导前存在一个有限的时间窗口，信号可被检测到。本文提出一个可解析处理的随机矩阵模型，重现了线性师生设置下梯度流中的这一现象。学习发生时，一个孤立特征值从噪声主体中分离，随后在过拟合阶段重新被吸收。关键机制是输入协方差的各向异性，导致学习动力学存在快慢方向。在两块协方差模型中，我们通过2×2杜辛方程推导出对称权重矩阵的全时变主体谱，并利用二阶行列式公式给出了一阶教师信号的异常值条件。这导致一种瞬态贝克-本·阿鲁斯-佩舍（BBP）转变：根据信号强度与协方差各向异性，教师信号峰可能永不出现、持续存在，或仅在中间时间段短暂出现后被重吸收。我们绘制了对应相图，并通过有限尺寸模拟验证了理论。结果提供了一个最小可解机制，解释了早期停止作为由各向异性和噪声驱动的瞬态谱效应。
原文摘要 · Abstract (English)
Empirical studies of trained models often report a transient regime in which signal is detectable in a finite gradient descent time window before overfitting dominates. We provide an analytically tractable random-matrix model that reproduces this phenomenon for gradient flow in a linear teacher--student setting. In this framework, learning occurs when an isolated eigenvalue separates from a noisy bulk, before eventually disappearing in the overfitting regime. The key ingredient is anisotropy in the input covariance, which induces fast and slow directions in the learning dynamics. In a two-block covariance model, we derive the full time-dependent bulk spectrum of the symmetrized weight matrix through a $2\times 2$ Dyson equation, and we obtain an explicit outlier condition for a rank-one teacher via a rank-two determinant formula. This yields a transient Baik-Ben Arous-Péché (BBP) transition: depending on signal strength and covariance anisotropy, the teacher spike may never emerge, emerge and persist, or emerge only during an intermediate time interval before being reabsorbed into the bulk. We map the corresponding phase diagrams and validate the theory against finite-size simulations. Our results provide a minimal solvable mechanism for early stopping as a transient spectral effect driven by anisotropy and noise.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。