低秩优化中梯度子空间无法稳定追踪，核心方向仅约39个可复现。
No Subspace to Track: Non-Identifiability and Optimizer State in Low-Rank Training
- 发现梯度子空间在不同小批量间差异大，非稳定可追踪对象
- 仅约39/128方向可复现，平均无法提升子空间清晰度
- 提出新优化器LDAdam，通过旋转不变性提升训练效果
内存高效的优化器如GaLore通过每T步重采样一个秩r的梯度子空间来训练大语言模型，假设该子空间缓慢漂移可被追踪。我们发现，除了一个小而可复现的核心外，并不存在这样的稳定子空间：从不同小批量计算的同一时刻顶部r子空间的差异，与相隔T步的估计值相当（Pythia-160M，r=128时为0.73 vs 0.74最大弦距离√(2r)）。这种看似旋转实则由估计噪声主导。该现象在四个模型族、三类架构、参数量70M至6.9B间均成立，且随规模增强；视觉变换器中较弱。仅有约39/128方向可在小批量间复现，平均化也无法恢复其余方向：在N倍平均下，梯度谱尾衰减为N^(-1/4)，而非纯噪声的N^(-1/2)，故无足够平均预算能定义清晰子空间。相反，若将每次刷新视为对Adam状态的坐标变换，则第二阶矩盲目传递会比最优无旋转估计差约(r-k*)/2倍，而一阶矩可精确通过旋转，即在各向同性梯度下最优线性映射，也是LDAdam所用规则。在1B参数模型上运行40k步（3种子），全量LDAdam在beta2=0.999下达到18.7困惑度，优于经最佳beta2修正后的未传输版本GaLore（19.3）；缩短二阶矩记忆至beta2=0.99有助于刷新型优化器，但对标准GaLore影响小，全秩对照反而逆转效果。一个可度量的事实——子空间不可识别性——阐明了为何GaLore有效、哪些修补有效，以及在信任低秩假设前应检查的关键指标：可复现秩k*。
原文摘要 · Abstract (English)
Memory-efficient optimizers such as GaLore train large language models by projecting gradients onto a rank-r subspace recomputed every T steps, assuming this subspace is a slowly drifting object that can be tracked. We show that beyond a small reproducible core, there is no such object. Two estimates of the top-r subspace computed at the same step from disjoint minibatches disagree as much as estimates computed T steps apart (0.73 vs 0.74 of the maximal chordal distance sqrt(2r), at Pythia-160M with r=128): the apparent rotation at each refresh is dominated by estimator noise. This holds across four model families in three architecture classes from 70M to 6.9B parameters, strengthening with scale, and more weakly in a vision transformer. Only ~39 of 128 directions are reproducible across minibatches, and averaging cannot recover the rest: under N-fold averaging the gradient's spectral tail shrinks as N^(-1/4) rather than the N^(-1/2) of pure noise, so no averaging budget makes the subspace well defined. What helps instead follows from treating each refresh as a change of coordinates for Adam's state. Carrying the second moment blindly is provably about (r-k*)/2 worse than the best rotation-blind estimator, while the first moment transports exactly through the rotation, the optimal linear map under isotropic gradients and the rule LDAdam uses. At 1B over 40k steps (3 seeds), full LDAdam reaches 18.7 perplexity at beta2=0.999, beating untransported GaLore after its best beta2 fix (19.3); shortening the second-moment memory to beta2=0.99 helps the refreshing optimizers, though for canonical GaLore the effect is small and a full-rank control reverses it. One measurable fact, subspace non-identifiability, clarifies why GaLore works, which patches work, and what to check before trusting a low-rank assumption: the reproducible rank k*.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。