用极简模型揭示深度网络训练中尖锐度上升的机制
Understanding Sharpness Dynamics in NN Training with a Minimalist Example: The Effects of Dataset Difficulty, Depth, Stochasticity, and More
- 构建每层一个神经元的线性网络，模拟真实训练中的尖锐度变化
- 发现数据难度、网络深度、优化器随机性等会加剧尖锐度上升
- 理论与实验结合，为理解训练稳定性提供新视角，适合研究者参考
使用每层仅含一个神经元的深层线性网络作为极简模型，研究梯度下降训练中神经网络尖锐度逐渐增加的现象（即渐进尖锐化）。该模型能有效复现近期实证研究中观察到的尖锐度动态，为理解训练过程提供简洁分析框架。理论上分析了数据集特性、网络深度、优化器随机性及学习率对尖锐度的影响，并在实际场景中验证了这些结论的可迁移性。研究揭示了网络深度、训练数据和优化器之间的复杂相互作用，深化了对神经网络训练中尖锐度演化的理解。
原文摘要 · Abstract (English)
When training deep neural networks with gradient descent, sharpness often increases -- a phenomenon known as progressive sharpening -- before saturating at the edge of stability. Although commonly observed in practice, the underlying mechanisms behind progressive sharpening remain poorly understood. In this work, we study this phenomenon using a minimalist model: a deep linear network with a single neuron per layer. We show that this simple model effectively captures the sharpness dynamics observed in recent empirical studies, offering a simple testbed to better understand neural network training. Moreover, we theoretically analyze how dataset properties, network depth, stochasticity of optimizers, and step size affect the degree of progressive sharpening in the minimalist model. We then empirically demonstrate how these theoretical insights extend to practical scenarios. This study offers a deeper understanding of sharpness dynamics in neural network training, highlighting the interplay between depth, training data, and optimizers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。