用熵力理论解释深度学习中的涌现现象。
Neural Thermodynamics: Entropic Forces in Deep and Universal Representation Learning
- 基于参数对称性和熵能景观，提出熵力理论。
- 揭示神经网络表示对齐与优化行为矛盾的根源。
- 适合研究深度学习机制与表示学习的学者。
随着深度学习和大语言模型中不断涌现新现象，理解其成因已成为迫切需求。本文提出一种严格的熵力理论，用于解析随机梯度下降（SGD）及其变体训练神经网络的学习动态。基于参数对称性与熵能景观理论，我们发现表示学习受随机性与离散时间更新引发的涌现熵力主导。这些力系统性打破连续参数对称性，同时保留离散对称性，导致一系列类似热力学系统能量均分的梯度平衡现象。该现象一方面解释了不同AI模型间神经表示的普遍对齐，并证明了柏拉图表示假设；另一方面调和了深度学习优化中对尖锐与平坦区域的看似矛盾的追求行为。理论与实验表明，熵力与对称性破缺的结合是理解深度学习涌现现象的关键。
原文摘要 · Abstract (English)
With the rapid discovery of emergent phenomena in deep learning and large language models, understanding their cause has become an urgent need. Here, we propose a rigorous entropic-force theory for understanding the learning dynamics of neural networks trained with stochastic gradient descent (SGD) and its variants. Building on the theory of parameter symmetries and an entropic loss landscape, we show that representation learning is crucially governed by emergent entropic forces arising from stochasticity and discrete-time updates. These forces systematically break continuous parameter symmetries and preserve discrete ones, leading to a series of gradient balance phenomena that resemble the equipartition property of thermal systems. These phenomena, in turn, (a) explain the universal alignment of neural representations between AI models and lead to a proof of the Platonic Representation Hypothesis, and (b) reconcile the seemingly contradictory observations of sharpness- and flatness-seeking behavior of deep learning optimization. Our theory and experiments demonstrate that a combination of entropic forces and symmetry breaking is key to understanding emergent phenomena in deep learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。