用神经坍缩解释模型训练后期的突变现象
Explaining Grokking and Information Bottleneck through Neural Collapse Emergence
- 通过神经坍缩机制分析训练后期表现突变
- 发现类内方差收缩是突变与信息瓶颈的关键
- 适用于研究深层模型训练动态的学者
深度神经网络的训练动态常违背预期，尽管它们构成了现代机器学习的基础。两个典型现象是：在训练损失已停滞后，测试性能突然显著提升（即grokking）；以及随着训练推进，模型逐步丢弃与预测无关的输入信息（即信息瓶颈）。然而，这些现象的机理及其关联仍不清楚。本文通过神经坍缩视角提出统一解释，该现象刻画了学习表示的几何特性。我们发现类内方差的收缩是grokking与信息瓶颈的关键因素，并将其与基于训练集定义的神经坍缩度量相关联。通过分析神经坍缩的动力学，揭示了拟合训练集与神经坍缩演进之间的不同时间尺度，解释了晚期现象的行为。最后，我们在多个数据集和架构上验证了理论结果。
原文摘要 · Abstract (English)
The training dynamics of deep neural networks often defy expectations, even as these models form the foundation of modern machine learning. Two prominent examples are grokking, where test performance improves abruptly long after the training loss has plateaued, and the information bottleneck principle, where models progressively discard input information irrelevant to the prediction task as training proceeds. However, the mechanisms underlying these phenomena and their relations remain poorly understood. In this work, we present a unified explanation of such late-phase phenomena through the lens of neural collapse, which characterizes the geometry of learned representations. We show that the contraction of population within-class variance is a key factor underlying both grokking and information bottleneck, and relate this measure to the neural collapse measure defined on the training set. By analyzing the dynamics of neural collapse, we show that distinct time scales between fitting the training set and the progression of neural collapse account for the behavior of the late-phase phenomena. Finally, we validate our theoretical findings on multiple datasets and architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。