用积分方程建模密集连接网络,证明其训练稳定性与收敛性。
Mathematical Modeling and Convergence Analysis of Deep Neural Networks with Dense Layer Connectivities in Deep Learning
- 将密集连接网络视为非线性积分方程,突破传统微分方程视角。
- 证明最优值收敛且极小化器存在子列收敛,理论支撑训练稳定性。
- 适合研究深度学习数学基础的学者,尤其关注模型收敛性者。
在深度学习中,密集连接层已成为深度神经网络(DNN)的关键设计原则,有助于高效信息流动并在多种应用中表现优异。本文从数学角度建模密集连接的DNN,并分析其在深层极限下的学习问题。为具广泛适用性,我们提出一种包含密集连接层和一般非局部特征变换(局部变换为特例)的框架,称为密集非局部(DNL)框架,涵盖标准DenseNets及其变体作为特例。在此框架下,密集连接网络被建模为非线性积分方程,不同于以往工作中常用的常微分方程视角。我们从最优控制角度研究相关训练问题,并证明了从离散网络学习问题到连续时间对应问题的收敛性。特别地,通过分段线性延拓与Γ-收敛分析,我们证明了最优值的收敛性以及极小化器的子列收敛性。结果为理解密集连接DNN提供了数学基础,并进一步表明此类架构可提升深度模型训练的稳定性。
原文摘要 · Abstract (English)
In deep learning, dense layer connectivity has become a key design principle in deep neural networks (DNNs), enabling efficient information flow and strong performance across a range of applications. In this work, we model densely connected DNNs mathematically and analyze their learning problems in the deep-layer limit. For a broad applicability, we present our analysis in a framework setting of DNNs with densely connected layers and general non-local feature transformations (with local feature transformations as special cases) within layers, which is called dense non-local (DNL) framework and includes standard DenseNets and variants as special examples. In this formulation, the densely connected networks are modeled as nonlinear integral equations, in contrast to the ordinary differential equation viewpoint commonly adopted in prior works. We study the associated training problems from an optimal control perspective and prove convergence results from the network learning problem to its continuous-time counterpart. In particular, we show the convergence of optimal values and the subsequence convergence of minimizers, using a piecewise linear extension and $Γ$-convergence analysis. Our results provide a mathematical foundation for understanding densely connected DNNs and further suggest that such architectures can offer stability of training deep models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。