提出DeVA框架，统一向量与矩阵优化的自适应梯度方法
Decoupling Variance and Scale-Invariant Updates in Adaptive Gradient Descent for Unified Vector and Matrix Optimization
- 将自适应更新分解为方差适应与尺度不变两部分，实现向量与矩阵优化的统一
- 在语言建模和图像分类任务中，比Muon、SOAP等方法减少约6.6%的令牌使用量
- 理论证明方差适应提升块状光滑性，加速收敛，适合大规模模型训练
自适应方法如Adam已成为大规模向量与欧氏优化的事实标准，因其具有坐标级自适应的二阶特性。近期，基于矩阵谱的优化器如Muon（Jordan et al., 2024b）展示了将权重矩阵视为矩阵而非长向量的优势。然而，这两者间的融合困难，许多自然推广难以实现，也无法简单将Adam的适应机制迁移至矩阵谱。为此，我们重新构建AdaGrad更新，并将其分解为方差适应项与尺度不变项。该解耦产生了一个名为DeVA（Decoupled Variance Adaptation）的框架，连接了基于向量的方差自适应与基于矩阵谱的优化，实现了从Adam到自适应谱下降的无缝过渡。在语言建模与图像分类任务上的广泛实验表明，DeVA持续优于当前最优方法如Muon和SOAP（Vyas et al., 2024），令牌使用量降低约6.6%。理论上，我们证明方差适应项有效提升块状光滑性，促进更快收敛。代码已开源：https://github.com/Tsedao/Decoupled-Variance-Adaptation
原文摘要 · Abstract (English)
Adaptive methods like Adam have become the $\textit{de facto}$ standard for large-scale vector and Euclidean optimization due to their coordinate-wise adaptation with a second-order nature. More recently, matrix-based spectral optimizers like Muon (Jordan et al., 2024b) show the power of treating weight matrices as matrices rather than long vectors. Linking these is hard because many natural generalizations are not feasible to implement, and we also cannot simply move the Adam adaptation to the matrix spectrum. To address this, we reformulate the AdaGrad update and decompose it into a variance adaptation term and a scale-invariant term. This decoupling produces $\textbf{DeVA}$ ($\textbf{De}$coupled $\textbf{V}$ariance $\textbf{A}$daptation), a framework that bridges between vector-based variance adaptation and matrix spectral optimization, enabling a seamless transition from Adam to adaptive spectral descent. Extensive experiments across language modeling and image classification demonstrate that DeVA consistently outperforms state-of-the-art methods such as Muon and SOAP (Vyas et al., 2024), reducing token usage by around 6.6\%. Theoretically, we show that the variance adaptation term effectively improves the blockwise smoothness, facilitating faster convergence. Our implementation is available at https://github.com/Tsedao/Decoupled-Variance-Adaptation
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。