发现深度网络几何不变性，解释了不同架构的泛化差异。
Geometric Inductive Biases of Deep Networks: The Role of Data and Architecture
- 提出输入空间曲率在特定方向上保持不变的几何不变性假说。
- ResNet因平均几何低秩导致曲率变化受限，影响平面分类泛化。
- 揭示架构决定几何演化，适合研究模型泛化与结构设计的人阅读。
本文提出几何不变性假说（GIH），认为神经网络在训练过程中，其输入空间曲率在某些架构相关的方向上保持不变。我们研究了一个高维空间中平面上的简单非线性二分类问题，发现与MLP不同，ResNet的泛化性能依赖于平面方向。为此，定义了神经网络的平均几何及其平均几何演化，作为模型输入-输出几何及其训练过程演化的紧凑架构依赖性总结。通过分析初始化时的平均几何演化，发现网络几何的变化由数据协方差投影到平均几何所决定。当平均几何为低秩时（如ResNet），几何仅在输入空间的子集内变化，从而产生架构依赖的输入空间曲率不变性，即GIH。最后，通过大量实验验证了GIH的影响及其与神经网络泛化的关系。
原文摘要 · Abstract (English)
In this paper, we propose the $\textit{geometric invariance hypothesis (GIH)}$, which argues that the input space curvature of a neural network remains invariant under transformation in certain architecture-dependent directions during training. We investigate a simple, non-linear binary classification problem residing on a plane in a high dimensional space and observe that$\unicode{x2014}$unlike MLPs$\unicode{x2014}$ResNets fail to generalize depending on the orientation of the plane. Motivated by this example, we define a neural network's $\textbf{average geometry}$ and $\textbf{average geometry evolution}$ as compact $\textit{architecture-dependent}$ summaries of the model's input-output geometry and its evolution during training. By investigating the average geometry evolution at initialization, we discover that the geometry of a neural network evolves according to the data covariance projected onto its average geometry. This means that the geometry only changes in a subset of the input space when the average geometry is low-rank, such as in ResNets. This causes an architecture-dependent invariance property in the input space curvature, which we dub GIH. Finally, we present extensive experimental results to observe the consequences of GIH and how it relates to generalization in neural networks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。