arXiv:2510.25000cs.LG2025-10被引 16

矩阵去相关优化器性能优越,关键在于自适应方差调整。

What Really Matters in Matrix-Whitening Optimizers?

  • 通过分解矩阵去相关优化器,发现其核心是自适应方差调整。
  • 自适应方差版本的优化器显著优于仅依赖符号下降的方法。
  • 低秩方差估计可在不损失性能前提下降低内存开销。

近期涌现的多种优化器均以不同方式近似相同的“矩阵去相关”变换。本文系统性地剖析这些优化器,旨在分离出影响性能的关键成分。在所有调优超参数下,各类矩阵去相关方法均显著优于元素级优化器(如Adam)。尽管矩阵去相关常与谱下降相关,但实验表明性能提升并非仅由精确的谱归一化解释——例如,SOAP每步提升最大,即便Muon更准确沿最陡谱方向下降。我们提出,矩阵去相关实现双重作用,而其中的方差自适应成分是被忽视的性能关键。实验显示,具备方差自适应的优化器始终优于符号下降型,包括改进版Muon。进一步消融实验表明,前瞻式近似效果不佳,但低秩方差估计可有效降低内存消耗且无性能损失。

原文摘要 · Abstract (English)

A range of recent optimizers have emerged that approximate the same "matrix-whitening" transformation in various ways. In this work, we systematically deconstruct such optimizers, aiming to disentangle the key components that explain performance. Across tuned hyperparameters across the board, all flavors of matrix-whitening methods reliably outperform elementwise counterparts, such as Adam. Matrix-whitening is often related to spectral descent -- however, experiments reveal that performance gains are *not explained solely by accurate spectral normalization* -- particularly, SOAP displays the largest per-step gain, even though Muon more accurately descends along the steepest spectral descent direction. Instead, we argue that matrix-whitening serves two purposes, and the variance adaptation component of matrix-whitening is the overlooked ingredient explaining this performance gap. Experiments show that variance-adapted versions of optimizers consistently outperform their sign-descent counterparts, including an adaptive version of Muon. We further ablate variance adaptation strategies, finding that while lookahead style approximations are not as effective, low-rank variance estimators can effectively reduce memory costs without a performance loss.

优化器矩阵去相关方差自适应深度学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。