揭示同质神经网络中解的嵌入规律,解释宽网如何继承窄网解。
Embedding principle of homogeneous neural network for classification problem
- 提出KKT点嵌入原理,用线性保距变换连接不同宽度网络的最优解
- 证明全连接与卷积网络在分拆神经元/通道时,解可精确嵌入
- 发现训练轨迹和最终方向保持映射关系,适合研究网络宽度影响
本文研究同质神经网络(包括全连接与卷积网络)最大间隔问题的Karush-Kuhn-Tucker(KKT)点。我们提出并形式化了 extbf{KKT点嵌入原则},证明一个网络的最大间隔问题$P_Φ$的KKT点可通过特定线性等距变换嵌入到更大网络的问题$P_{\tildeΦ}$的KKT点中。该原则在全连接网络的神经元分裂与卷积网络的通道分裂下均被严格证明成立。进一步,我们将静态嵌入与平滑损失下的梯度流训练动态相联系:从适当映射起点出发的轨迹始终维持映射关系,且方向的ω-极限集也相应映射,从而在方向收敛时动态保持与KKT方向的一致性。通过多个实验验证了轨迹的保留性。研究结果为理解网络宽度、参数冗余及不同规模网络间解的结构关联提供了新视角。
原文摘要 · Abstract (English)
In this paper, we study the Karush-Kuhn-Tucker (KKT) points of the associated maximum-margin problem in homogeneous neural networks, including fully-connected and convolutional neural networks. In particular, We investigates the relationship between such KKT points across networks of different widths generated. We introduce and formalize the \textbf{KKT point embedding principle}, establishing that KKT points of a homogeneous network's max-margin problem ($P_Φ$) can be embedded into the KKT points of a larger network's problem ($P_{\tildeΦ}$) via specific linear isometric transformations. We rigorously prove this principle holds for neuron splitting in fully-connected networks and channel splitting in convolutional neural networks. Furthermore, we connect this static embedding to the dynamics of gradient flow training with smooth losses. We demonstrate that trajectories initiated from appropriately mapped points remain mapped throughout training and that the resulting $ω$-limit sets of directions are correspondingly mapped, thereby preserving the alignment with KKT directions dynamically when directional convergence occurs. We conduct several experiments to justify that trajectories are preserved. Our findings offer insights into the effects of network width, parameter redundancy, and the structural connections between solutions found via optimization in homogeneous networks of varying sizes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。