arXiv:2502.04664cs.LGmath.OC2025-02NeurIPS被引 32

揭示多分类下矩阵优化算法的隐式偏好,解释为何不同方法收敛到不同解。

Implicit Bias of Spectral Descent and Muon on Multiclass Separable Data

  • 通过分析p-范数梯度下降,发现算法会趋向最大化分类矩阵p-范数的边界解。
  • Spectral Descent和Muon等算法在多分类中收敛到谱范数最大边界解,有明确收敛速率。
  • 理论可推广至Adam等预条件方法,适合研究优化算法泛化机制的研究者阅读。

针对过参数化模型,不同基于梯度的方法虽均能实现零训练误差,却收敛于具有不同泛化性质的解。本文首次完整刻画了多分类线性分类任务中交叉熵损失下p-范数归一化最陡下降(NSD)与动量最陡下降(NMD)算法的隐式优化偏差。关键理论贡献在于证明这些算法收敛于最大化分类矩阵p-范数边界的解,并给出了确定的收敛速率。该结果涵盖重要特例:Spectral Descent和Muon收敛于谱范数最大边界解。核心洞察在于,通过对所有范数相对于最大范数及其对偶和范数的自然序关系,可将一般元素与Schatten p-范数分析简化为最大范数下的分析。对于最大范数下降情形,进一步引入预条件分析,表明Adam收敛于矩阵最大范数解。结果表明,相较于二分类,多分类线性设置提供了研究矩阵参数优化算法隐式偏差最清晰的框架。

原文摘要 · Abstract (English)

Different gradient-based methods for optimizing overparameterized models can all achieve zero training error yet converge to distinctly different solutions inducing different generalization properties. We provide the first complete characterization of implicit optimization bias for p-norm normalized steepest descent (NSD) and momentum steepest descent (NMD) algorithms in multi-class linear classification with cross-entropy loss. Our key theoretical contribution is proving that these algorithms converge to solutions maximizing the margin with respect to the classifier matrix's p-norm, with established convergence rates. These results encompass important special cases including Spectral Descent and Muon, which we show converge to max-margin solutions with respect to the spectral norm. A key insight of our contribution is that the analysis of general entry-wise and Schatten p-norms can be reduced to the analysis of NSD/NMD with max-norm by exploiting a natural ordering property between all p-norms relative to the max-norm and its dual sum-norm. For the specific case of descent with respect to the max-norm, we further extend our analysis to include preconditioning, showing that Adam converges to the matrix's max-norm solution. Our results demonstrate that the multi-class linear setting, which is inherently richer than the binary counterpart, provides the most transparent framework for studying implicit biases of matrix-parameter optimization algorithms.

优化算法隐式偏差多分类谱范数

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。