arXiv:2608.21024cs.LG2026-08

提出分离权重大小与方向更新的新优化器,提升模型训练效率与稳定性。

RODE: A Radial-Orthogonal Decoupled Engine for Optimization

  • 将权重的大小和方向分别用独立规则更新,避免相互干扰。
  • 在1.5B和9B模型上均降低损失,最终权重范数从11964降至2183。
  • 适合追求更稳定、可控训练过程的语言和图像模型优化场景。

现代神经网络训练越来越多地使用矩阵感知优化器,但其条件化的矩阵步长通常直接加到权重上,同时改变权重的模长和方向。这种耦合效应至关重要:当前的模长决定方向运动,而方向学习又可能驱动模长增长,从而影响后续步骤。我们提出RODE,为径向和方向分量设计独立的更新规则和步长。RODE通过标量径向规则显式更新矩阵Frobenius范数,其方向通道则在切空间中进行Newton–Schulz条件化更新。控制实验显示,直接控制模长和方向更新均带来性能提升。在两个语言建模和两个图像分类任务中,RODE在所有直接对比中优于两种Muon变体,并最终获得更低的全模型范数。在15亿参数规模下,使用来自Qwen2风格语言模型搜索的直接学习率迁移,RODE将损失从4.145降至3.346,最终全局范数从11964降至2183(相比Muon RMS);固定半径的RODE进一步提升性能。对于Qwen3.5-9B全参数微调,所有六种优化器使用相同调优预算与正式训练评估设置;RODE在全部四项评估任务上超越两种Muon变体,在GSM8K和MATH-500上取得最高平均表现。因此,解耦径向与方向动力学提供了一种更有效、更可控的矩阵优化方法。

原文摘要 · Abstract (English)

Modern neural network training increasingly uses matrix-aware optimizers, yet their conditioned matrix step is typically added directly to the weight, jointly changing its norm and direction. This interaction matters because the current norm determines angular motion, while directional learning can drive norm growth and thereby alter later steps. We introduce RODE, which gives the radial and directional components separate update rules and step sizes. RODE explicitly updates the matrix Frobenius norm through a scalar radial rule, while its directional channel performs Newton--Schulz-conditioned updates in the tangent space. Controlled GPT-2 interventions show gains from both direct norm control and RODE's directional update. Across two language-modeling and two image-classification tasks, RODE outperforms both Muon variants in every direct comparison and ends with lower full-model norms. At 1.5B scale, using the learning rate transferred directly from the Qwen2-style LM sweep, RODE lowers loss from 4.145 to 3.346 and final global norm from 11964 to 2183 relative to Muon RMS, with fixed-radius RODE improving further. For Qwen3.5-9B full-parameter fine-tuning, all six optimizers use the same tuning budget and the same formal-training and evaluation settings; RODE outperforms both Muon variants on all four evaluation tasks and attains the highest mean on GSM8K and MATH-500. Thus, decoupling radial and directional dynamics offers a more effective and controllable approach to matrix optimization.

优化器模型训练深度学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。