通过保持参数对称性,提升优化器在复杂模型上的稳定性与可读性。
Dead-Direction Conditioners: Gauge-Equivariant Preconditioning for Deep Networks

- 构建基于对称性不变量的自适应预条件器DDC,使优化轨迹保持在商空间上
- 在语言模型上,验证集与训练集损失差仅为0.67,远优于AdamW的5.88
- 适合追求优化精度、理解模型内在结构的研究者
深度网络的损失函数对参数的连续对称性(如logit平移、ReLU缩放、LayerNorm尺度、每头注意力旋转)保持不变。Adam的逐坐标预条件器会沿对称轨道漂移,使优化轨迹偏离优化所处的对称商空间,模糊了该空间中可读的奇异学习率。我们提出死方向调节器DDC,将基础优化器升维为G-等变形式:它在G-不变度量的轨道分解中调整优化器状态,使轨迹始终是商空间Θ/G上的预条件梯度流。该构造兼容四种架构规范(交叉熵平移、ReLU/SwiGLU缩放、LayerNorm/RMSNorm尺度、匹配RoPE的每头O(d_head)旋转),在Adam基础上实现精确等变,并可通过等变正交化器与Muon组合。尊重对称性不仅改变了优化到达的最小值,也使其在该点可测量。在过拟合后的语言模型上,DDCAdam避免了AdamW的过训练崩溃,保持验证-训练损失差0.67(而AdamW为5.88),并在32/65层-观测单元中读出死方向率(而AdamW仅7个)。从零训练的视觉变换器达到更低验证损失(1.71对2.12),同时压缩了冗余前馈容量。在Muon基础上,当旋转规范完全组合时,DDCMuon在11个种子中成功收敛10个(深度24),而普通Muon从未达到。预条件器内嵌对称性,使网络找到更锐化的极小值,并将其几何转化为轨迹可测的信号。
原文摘要 · Abstract (English)
A deep network's loss is invariant to continuous symmetries of its parameters: the logit shift, the ReLU rescaling, the LayerNorm scale, the per-head attention rotation. Adam's per-coordinate preconditioner drifts along each symmetry orbit, which pulls the trajectory off the symmetry quotient where the optimization lives and blurs the singular-learning rate the quotient makes readable. We build DDC, a Dead-Direction Conditioner that lifts a base optimizer into a $G$-equivariant one: it conditions the optimizer's state in the orbit decomposition of a $G$-invariant metric, so the trajectory stays a preconditioned gradient flow on the quotient $\barΘ= Θ/G$. The construction carries four architectural gauges (cross-entropy shift, ReLU and SwiGLU rescaling, LayerNorm and RMSNorm scale, and a per-head $O(d_{\rm head})$ attention rotation matched to RoPE), proves exactly equivariant on an Adam base, and composes with a Muon base through a gauge-equivariant orthogonaliser. Respecting the symmetry changes both the minimum the optimizer reaches and what it leaves measurable there. On a language model trained past the point of fit, DDCAdam resists the over-training collapse AdamW falls into, holding a validation-train loss gap of 0.67 against 5.88, and reads the dead-direction rate in 32 of 65 layer-by-observable cells where AdamW reads it in 7. A vision transformer trained from scratch reaches lower validation loss (1.71 against 2.12) while compressing spare feed-forward capacity a matched AdamW leaves intact. On a Muon base, where the rotation gauge composes exactly, DDCMuon groks ten of eleven seeds at depth 24 that a plain Muon never reaches. Built into the optimizer, a network's gauge symmetry sharpens the minimum it finds and turns that minimum's geometry into something the trajectory can measure.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。