用平均场理论统一解析对角线神经网络的训练动态。
Precise Dynamics of Diagonal Linear Networks: A Unifying Analysis by Dynamical Mean-Field Theory
- 基于动力学平均场理论，建立高维训练的低维有效模型。
- 揭示损失收敛速率与泛化能力间的权衡关系。
- 适用于研究复杂训练现象的理论分析，适合机器学习研究者。
对角线线性网络（DLNs）是一种可解析的模型，能捕捉神经网络训练中的多种非平凡行为，如初始化依赖解和增量学习。这些现象通常被孤立研究，整体动态理解不足。本文利用动力学平均场理论（DMFT），推导出高维下渐近梯度流动态的低维有效过程。分析该有效过程揭示了新的见解，包括损失收敛速率及其与泛化性能的权衡，并系统重现了以往观察到的多种现象。这些发现深化了对DLN动态的理解，展示了DMFT在分析高维神经网络学习动态中的有效性。
原文摘要 · Abstract (English)
Diagonal linear networks (DLNs) are a tractable model that captures several nontrivial behaviors in neural network training, such as initialization-dependent solutions and incremental learning. These phenomena are typically studied in isolation, leaving the overall dynamics insufficiently understood. In this work, we present a unified analysis of various phenomena in the gradient flow dynamics of DLNs. Using Dynamical Mean-Field Theory (DMFT), we derive a low-dimensional effective process that captures the asymptotic gradient flow dynamics in high dimensions. Analyzing this effective process yields new insights into DLN dynamics, including loss convergence rates and their trade-off with generalization, and systematically reproduces many of the previously observed phenomena. These findings deepen our understanding of DLNs and demonstrate the effectiveness of the DMFT approach in analyzing high-dimensional learning dynamics of neural networks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。