arXiv:2607.09967cs.LGcs.AI2026-07

通过曲线权重空间重参数化,让神经网络优化更快更稳。

Learning in Curved Weight Space:Exponential-Linear Weight Reparameterization for Improved Optimization

论文配图:Learning in Curved Weight Space:Exponential-Linear Weight Reparameterization for Improved Optimization
图 1 · 摘自论文原文
  • 用指数-线性路径重构权重空间,使更新更符合相对变化规律。
  • 在九种模型配置下,训练步数减少1.32至1.49倍,大模型收益显著。
  • 适合追求高效训练的深度学习研究者,尤其关注优化器改进者。

许多神经网络操作具有乘法性质而非加法:减半或加倍权重幅度在相对意义上等价,但线性更新时需不同优化距离。自适应优化器如Adam虽按坐标归一化更新,但步长仍为加法;不同量级权重接受相似绝对变化,导致相对扰动差异大。我们提出 extbf{ exttt{Exponential-Linear Weight Reparameterization}}( extbf{ exttt{ELWR}}),一种结合符号感知对称指数路径与类恒等线性路径的权重重参数化方法。指数路径在小权重时近似线性,大权重时逐渐弯曲;对数空间的加法更新映射到有效权重空间的幅度比例变化。线性路径提供直接通路,假设可稳定优化;可学习的尺度、曲率和偏移参数控制路径平衡与指数路径弯曲程度。该设计形成弯曲参数空间几何,实证提升损失下降速度。我们还发现一种有用的‘不匹配初始化’:原始权重按对称变换匹配Xavier统计,但训练使用非对称前向变换,正权重保持原强度,负权重幅度变小;小模型消融实验显示其改善早期优化,可能起到对称破缺作用。在OpenWebText上训练Transformer,九种宽度×深度配置下, exttt{ELWR}以1.32–1.49倍较少步骤达到相同验证损失,宽度越大收益越明显。

原文摘要 · Abstract (English)

Many neural networks operations have a multiplicative nature rather than additive: halving or doubling a norm are analogous relatively but require unequal optimization distances when taking linear steps. Adaptive optimizers such as Adam normalize updates per coordinate, but update steps remain additive; weights with very different magnitudes receive similarly sized absolute changes, producing very different relative perturbations. We introduce \textbf{\method} (\textbf{\methodshort}), a weight reparameterization for neural networks that combines a sign-aware symmetric-exponential pathway with an identity-like linear pathway. The symmetric-exponential pathway is near-linear for small raw weights but increasingly curved at larger magnitudes. Additive updates in logarithmic space map to magnitude-proportional changes in effective weight space. The linear pathway provides a direct route through the transform that we hypothesize stabilizes optimization, while learnable scale, curvature, and offset parameters control balance between pathways and the curvature of the exponential pathway. These components create a curved parameter-space geometry that empirically improves speed of loss descent over standard linear parameterization. We also identify a useful \emph{mismatched initialization}: raw weights are chosen so a symmetric version of the transform matches Xavier statistics, but training uses an asymmetric forward transform that leaves positive weights at full strength while making negative weights smaller in magnitude; in small-model ablations, this improves early optimization and may act as a form of symmetry breaking. We train transformers on OpenWebText over nine width$\times$depth configurations, \methodshort reaches matched validation loss in 1.32--1.49$\times$ fewer training steps, with the largest widths seeing the biggest gains.

优化器权重重参数化Transformer

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。