优化器选择影响神经坍缩的出现,权重衰减耦合是关键。
Optimizer choice matters for the emergence of Neural Collapse
- 提出新指标NC0,理论证明自适应优化器中解耦权重衰减阻碍神经坍缩
- 发现SGD和带动量的优化器能加速神经坍缩,而AdamW则抑制该现象
- 3900次实验验证理论,适合研究训练动态与优化器设计的学者
神经坍缩(Neural Collapse, NC)指深度神经网络在训练末期表现出高度对称的几何结构。尽管普遍发生,其理论理解仍有限,现有分析多忽略优化器的影响,认为NC在所有优化方法下均普遍存在。本文挑战这一假设,证明优化器选择对NC的出现至关重要。为克服传统NC指标难以理论分析的问题,我们引入新诊断指标NC0,其趋近零是NC成立的必要条件。理论表明,在自适应优化器中采用解耦权重衰减(如AdamW)时,NC无法出现。具体而言,我们证明了SGD、耦合权重衰减的SignGD(Adam特例)与解耦权重衰减的SignGD(AdamW特例)在NC0动态上呈现定性差异。此外,首次揭示动量在训练损失收敛后仍能加速神经坍缩。最后,通过3,900次跨数据集、架构、优化器与超参数的实验,验证了理论结论。本工作首次从理论上解释了优化器依赖的神经坍缩现象,凸显权重衰减耦合对优化器隐式偏见的重要影响。
原文摘要 · Abstract (English)
Neural Collapse (NC) refers to the emergence of highly symmetric geometric structures in the representations of deep neural networks during the terminal phase of training. Despite its prevalence, the theoretical understanding of NC remains limited. Existing analyses largely ignore the role of the optimizer, thereby suggesting that NC is universal across optimization methods. In this work, we challenge this assumption and demonstrate that the choice of optimizer plays a critical role in the emergence of NC. The phenomenon is typically quantified through NC metrics, which, however, are difficult to track and analyze theoretically. To overcome this limitation, we introduce a novel diagnostic metric, NC0, whose convergence to zero is a necessary condition for NC. Using NC0, we provide theoretical evidence that NC cannot emerge under decoupled weight decay in adaptive optimizers, as implemented in AdamW. Concretely, we prove that SGD, SignGD with coupled weight decay (a special case of Adam), and SignGD with decoupled weight decay (a special case of AdamW) exhibit qualitatively different NC0 dynamics. Also, we show the accelerating effect of momentum on NC (beyond convergence of train loss) when trained with SGD, being the first result concerning momentum in the context of NC. Finally, we conduct extensive empirical experiments consisting of 3,900 training runs across various datasets, architectures, optimizers, and hyperparameters, confirming our theoretical results. This work provides the first theoretical explanation for optimizer-dependent emergence of NC and highlights the overlooked role of weight-decay coupling in shaping the implicit biases of optimizers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。