揭示神经网络近似中功能等价与几何多样性的共现规律。
Functional Equivalence and Geometric Diversity in Neural Network Approximations: An Empirical Characterization
- 通过分析损失函数海森矩阵谱，量化参数空间低维性。
- 发现大量功能相同但结构迥异的网络，具显著结构冗余。
- 提出基于简约性与可推断效率的模型选择准则。
通用逼近定理指出，单隐层神经网络即可在紧集上任意逼近任意连续一元函数。然而，此类网络表示不唯一，引发实际可辨识性问题。本文通过分析若干初等数学函数的神经网络近似，研究其功能等价性与几何多样性。涵盖噪声与无噪声条件下的单层网络与多层感知机。除网络容量外，还从笨拙性（sloppiness）视角考察几何特性，利用代价函数海森矩阵特征谱与有效秩量化参数空间维度。研究发现，存在大量功能不可区分但几何结构各异的网络，均呈现低有效秩与结构性冗余。最终提出一种基于简约性、可估计性与推断效率的模型选择准则。
原文摘要 · Abstract (English)
The Universal Approximation Theorem states that a neural network with a single hidden layer is sufficient to approximate any continuous univariate function on a compact domain to arbitrary error. However, the uniqueness of such neural network representations is not guaranteed, raising questions about practical identifiability. In this work, we address this concern by analyzing functional equivalence and geometric diversity of neural network approximations to a few elementary mathematical functions. The analysis includes an extensive study of single-layer neural networks and multilayer perceptrons under noisy and noise-free conditions. Beyond just network capacity, we study the geometric properties through the lens of sloppiness, characterized by the eigen spectrum of the Hessian of the cost function and the effective rank to quantify the dimensionality of parameter space. The study reveals large equivalence classes of functionally indistinguishable yet geometrically diverse networks that consistently exhibit low effective rank and structural redundancy. Finally, a model select criterion is proposed for identifying optimal models based on parsimony, ease of estimation, and inference efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。