研究FBS网络的深层极限学习收敛性与稳定性,为神经网络设计提供理论支撑。
Deep-layer limit and stability analysis of the basic forward-backward-splitting induced network (II): learning problems
- 基于FBS算法构建深层网络,分析其学习问题的数学性质。
- 证明网络训练收敛到深层极限系统的最优解,满足Γ-收敛性。
- 揭示学习参数对扰动的稳定性,适合优化与深度学习研究者参考。
从迭代优化方法和数值常/偏微分方程(ODE/PDE)中衍生出的深度展开神经网络,在过去十年中受到数据科学领域的广泛关注。其中,大量重要网络架构源自基本前向-后向分裂(FBS)算法。本文继续研究由原始FBS算法通过直接参数松弛构造的基本FBS诱导网络。在前期前向系统分析的差分/微分包含框架基础上,本文探讨了相应学习问题的若干理论性质。在较弱假设下,建立了基本FBS诱导网络训练问题向深层极限系统学习问题的普遍收敛性,表明训练参数的任意聚点均为深层极限系统学习问题的解,体现了Γ-收敛性。同时给出了学习问题扰动稳定性的定性分析,并通过一个简单数值实验验证了主要收敛结果。
原文摘要 · Abstract (English)
Deep unfolding neural networks derived from iterative optimization schemes and numerical ordinary/partial differential equations (ODEs/PDEs) have attracted much attention in data science over the last decade. Therein, numerous important network architectures were constructed from the basic forward-backward-splitting (FBS) algorithm. In this paper, we continue our research on the most basic FBS-induced network, an architecture unrolled from the original FBS algorithm by incorporating direct parameter relaxations. Following the difference/differential inclusion formulations in our previous forward system analyses, we here consider some theoretical aspects of corresponding learning problems. Under some mild assumptions, we establish a general convergence property of the training problem of the basic FBS-induced network to the learning problem of the deep-layer limit system, implying a $Γ$-convergence argument showing that any cluster point of the optimal learning parameters for the network is a solution to the learning problem of the deep-layer limit system. A qualitative analysis of perturbation stabilities of these learning problems is also presented. A simple numerical experiment is conducted to validate our main general convergence result.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。