用统计物理方法揭示深度网络泛化机制,突破无限宽假设
Statistical Physics of Deep Neural Networks: Generalization Capability, Beyond the Infinite Width, and Feature Learning
- 从数据平均视角推导泛化上界,仅依赖最后一层规模
- 在有限宽度下获得单隐层回归的泛化误差闭式解
- 揭示深层网络与学生t过程的联系,适合理论研究者
深度神经网络在众多任务中表现优异,常超越人类水平,但其内部机制仍如黑箱般难以理解。本文借助统计物理思想,从三个互补角度解析DNN:首先通过数据平均,推导出仅依赖最后一层规模的泛化渐近界,揭示深层结构的信息处理差异;其次采用数据依赖视角,在有限宽度热力学极限下,得到单隐层网络(回归任务)的泛化误差闭式表达式,给出更深架构的近似配分函数,并发现其与学生t过程的关联;最后从任务显式视角,分析网络对受控数据集的响应,探究其是否真正内化数据结构(趋近教师模型)而非仅记忆。该工作通过统计物理与机器学习的融合,深化了对DNN内在行为的理解。
原文摘要 · Abstract (English)
Deep Neural Networks (DNNs) excel at many tasks, often rivaling or surpassing human performance. Yet their internal processes remain elusive, frequently described as "black boxes." While performance can be refined experimentally, achieving a fundamental grasp of their inner workings is still a challenge. Statistical Mechanics has long tackled computational problems, and this thesis applies physics-based insights to understand DNNs via three complementary approaches. First, by averaging over data, we derive an asymptotic bound on generalization that depends solely on the size of the last layer, rather than on the total number of parameters -- revealing how deep architectures process information differently across layers. Second, adopting a data-dependent viewpoint, we explore a finite-width thermodynamic limit beyond the infinite-width regime. This leads to: (i) a closed-form expression for the generalization error in a finite-width one-hidden-layer network (regression task); (ii) an approximate partition function for deeper architectures; and (iii) a link between deep networks in this thermodynamic limit and Student's t-processes. Finally, from a task-explicit perspective, we present a preliminary analysis of how DNNs interact with a controlled dataset, investigating whether they truly internalize its structure -- collapsing to the teacher -- or merely memorize it. By understanding when a network must learn data structure rather than just memorize, it sheds light on fostering meaningful internal representations. In essence, this thesis leverages the synergy between Statistical Physics and Machine Learning to illuminate the inner behavior of DNNs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。