通过几何等价性，用超球优化器复现其他优化器的训练轨迹。
HyperTransfer: Understanding the Equivalence between Base Optimizer and Hyperball
- 基于参数方向更新的超球优化器可等效复现传统优化器动态。
- 仅需初始值和学习率调度，即可精确还原目标优化器的损失轨迹。
- 适用于任意网络结构，尤其适合理解优化器行为机制的研究者。
超球优化器通过约束参数范数并仅更新其方向，形成与传统基优化器（同时更新范数与方向）截然不同的神经网络优化范式。然而我们发现，对于尺度不变网络，这两种范式在动态上是等价的。基于此等价性，本文提出 HyperTransfer，仅利用目标基优化器的初始化与学习率调度，构建一个能复现其训练动态的超球优化器，无需实际运行目标优化器。我们进一步推导了逆映射，并将框架扩展至非尺度不变网络。实验表明，HyperTransfer 及其逆映射产生的损失轨迹几乎与目标优化器完全一致，表明超球优化的动态主要受有效学习率调度与优化器状态所支配。
原文摘要 · Abstract (English)
Hyperball optimizers constrain parameter norms and update only their directions, establishing a distinct paradigm for neural network optimization. Although this geometry appears fundamentally different from that of conventional Base Optimizers, which update both parameter norms and directions, we show that the two paradigms are dynamically equivalent for scale-invariant networks. Building on this equivalence, we propose HyperTransfer, which constructs a Hyperball optimizer that reproduces the dynamics of a target Base Optimizer using only its initialization and learning-rate schedule, without running the target optimizer itself. We further derive the inverse mapping and extend the framework to non-scale-invariant networks. Experiments show that both HyperTransfer and the inverse mapping produce loss trajectories nearly identical to those of their targets, suggesting that Hyperball dynamics are governed primarily by the induced effective learning-rate schedule and optimizer state.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。