arXiv:2606.16371cs.LG2026-06被引 1

用历史信息复用加速优化器极分解,提升训练效率。

CacheMuon: Using Temporal Preconditioning To Approximate Polar Factor

  • 利用动量矩阵的时序相关性,复用过往极分解结果
  • 在语言与视觉任务上减少约50%以上正交化计算量
  • 可调节缓存策略,兼顾精度与训练速度

Muon 是一种基于动量矩阵极分解的优化器,在多种训练场景中表现优异。其核心是使用 Newton-Schulz 迭代计算极分解,虽避免了精确奇异值分解的开销,但每步迭代仍代价高昂。由于动量矩阵随训练平滑变化,其对应的极分解具有强时序相关性。本文提出 CacheMuon,一种基于时间预处理的方法,通过重用先前优化步骤的信息来近似当前步骤的极分解,从而减少重复正交化计算。我们将 CacheMuon 视为不精确的 Muon 更新,其误差由新求解器误差和缓存陈旧度控制。实验表明,CacheMuon 在语言模型和视觉任务上,采用保守阈值时几乎等效于原版 Muon,同时显著降低正交化浮点运算量;更激进的设置则带来更大算力节省,仅伴随小幅验证性能下降。

原文摘要 · Abstract (English)

Muon is an optimizer that computes updates using the polar factor of the momentum matrix and has shown strong empirical performance across a range of training settings. A key component of Muon is the Newton-Schulz iteration used to compute this polar factor. Although this avoids the cost of an exact singular value decomposition, it remains expensive in practice because it is applied at every optimization step. At the same time, the momentum matrix changes smoothly over training, suggesting strong temporal correlation in the corresponding polar factors. In this paper, we exploit this structure and propose CacheMuon, a temporal preconditioning method that reuses information from previous optimization steps to approximate the polar factor at the current step. This reduces redundant orthogonalization computation across iterations. We analyze CacheMuon as an inexact Muon update, with error controlled by fresh-solver error and cache staleness. Empirically, CacheMuon provides a controllable quality-efficiency frontier: conservative thresholds closely match fresh Muon on language-model and vision training while reducing orthogonalization FLOPs, whereas more aggressive thresholds yield larger arithmetic savings at the cost of modest validation-quality degradation.

优化器极分解缓存训练加速

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。