Adam优化器会破坏矩阵分解的对称性,导致高秩解;而梯度下降能保持低秩偏好。
The Loss Does Not See the Basis, but Adam Does
- 通过分析损失函数的规范对称性,解释为何不同优化器对低秩解有不同倾向
- 在矩阵感知任务中,梯度下降比Adam误差降低43%-44%,尤其在低秩场景下优势明显
- 适用于研究优化器隐式偏置、模型结构与初始化敏感性的研究人员
在因子化模型 $W = UV^ op$ 上进行梯度下降时,其隐式偏向于低秩解,而从相同小初始化出发的Adam则不然。我们追溯差异源于损失函数的规范对称性,即 $(U, V) \mapsto (UQ, VQ)$ 下不变性。只有具备规范等变性质的优化器才能继承梯度流的低秩机制,这一条件虽必要但不充分。梯度下降、动量法、'共享标量'Adam、Muon和Shampoo满足该性质,而Adam、RMSProp及其它坐标级方法不满足。一个结构定理指出,无记忆等变规则恰好是格拉姆决定的左预处理算子,转移定理将梯度流的路径性质传递至共标量流。我们在九种更新规则下对欠定矩阵感知任务进行排序,以恢复误差对比真实解。一个从坐标级到共享标量预处理的单参数族逐步恢复低秩偏倚,揭示各向异性为根本原因。'谱调度'调和了关于Muon的矛盾报告:等速率更新可精确恢复低秩目标,但随着谱尾增长,优势消失。在Transformer中,Adam在第一步就分离了两个规范等价初始化,而等变优化器保持浮点精度,最终每头不变量 $W_Q^\top W_K$ 的相对弗罗贝尼乌斯距离相差56%,此差距无法通过每头旋转弥合。在两个高光谱数据集上,当训练损失相同时,梯度下降在最低采样密度下将保留误差降低43-44%,且在更低有效秩条件下表现更优。因此,基底选择并非调参细节,而是决定优化器选取何种插值解的关键决策。
原文摘要 · Abstract (English)
Gradient descent on a factored model $W = UV^\top$ is implicitly biased toward low-rank solutions, while Adam, starting from the same small initialization, is not. We trace the difference to the gauge symmetry of the loss, its invariance under $(U, V) \mapsto (UQ, VQ)$. Gradient flow's low-rank mechanism is available to an optimizer only if that optimizer is gauge-equivariant, a condition necessary for the transfer but not sufficient for low-rank recovery. Gradient descent, momentum, "shared-scalar" Adam, Muon, and Shampoo satisfy it. Adam, RMSProp, and the other coordinate-wise methods do not. A structure theorem characterizes the memoryless equivariant rules as exactly the Gram-determined left preconditioners, and a transfer theorem carries gradient flow's pathwise properties to common-scalar flows. We then sort nine update rules on underdetermined matrix sensing by recovery error against the planted ground truth. A one-parameter family from coordinate-wise to shared-scalar preconditioning restores the bias monotonically, isolating anisotropy as the cause. A "spectral schedule" reconciles two opposing reports about Muon: equal-rate updates recover exactly low-rank targets but lose their edge as the spectral tail grows. In transformers, Adam separates two gauge-equivalent initializations at the first step, where the equivariant optimizers stay at float precision, and ends with the per-head invariants $W_Q^\top W_K$ 56% apart in relative Frobenius distance, a gap no per-head rotation can close. On two hyperspectral datasets at matched training loss, gradient descent cuts held-out error by 43-44% at the lowest sampling density, and at lower effective rank. Basis choice is therefore not a tuning detail but a decision about which interpolant the optimizer selects.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。