神经网络训练中神经元会自发分组，影响模型泛化与推理能力。
An overview of condensation phenomenon in deep learning
- 训练时同层神经元输出趋于相似，形成凝聚簇。
- 凝聚簇数量随训练单调增长，小初始化或Dropout可加速此过程。
- 揭示了模型泛化与推理能力的内在机制，适合研究模型内部动态者阅读。
本文综述了深度学习中一种常见现象——凝聚：在神经网络的非线性训练过程中，同一层内的神经元倾向于凝聚成输出相似的群体。实证观察表明，同层神经元的凝聚簇数量通常随训练进程单调增加。采用小权重初始化或使用Dropout优化的神经网络更易促进该凝聚过程。我们从训练动力学和损失曲面结构两个角度探讨了凝聚现象的潜在机制。该现象为理解神经网络的泛化能力提供了重要视角，并与基于Transformer的语言模型更强的推理能力相关联。
原文摘要 · Abstract (English)
In this paper, we provide an overview of a common phenomenon, condensation, observed during the nonlinear training of neural networks: During the nonlinear training of neural networks, neurons in the same layer tend to condense into groups with similar outputs. Empirical observations suggest that the number of condensed clusters of neurons in the same layer typically increases monotonically as training progresses. Neural networks with small weight initializations or Dropout optimization can facilitate this condensation process. We also examine the underlying mechanisms of condensation from the perspectives of training dynamics and the structure of the loss landscape. The condensation phenomenon offers valuable insights into the generalization abilities of neural networks and correlates to stronger reasoning abilities in transformer-based language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。