揭示了深层神经网络中关键特征空间的数学本质。
Approximating Simple ReLU Networks based on Spectral Decomposition of Fisher Information
- 基于费舍尔信息谱分解，识别出主导特征空间
- 前三个特征空间贡献97.7%的迹，与参数量无关
- 对应阶数不超过2的球谐函数，适用于理论分析
研究了具有随机隐藏权重的两层神经网络中费舍尔信息矩阵的性质。已知这些网络的特征值分布高度集中在少数特征空间上，其中前三个特征空间的特征值之和占费舍尔信息矩阵迹的97.7%，且与参数数量无关。本文确定了这些主要特征空间所对应的函数空间，发现其由阶数不超过2的球谐函数构成。该结果与神经正切核的Mercer分解相关。
原文摘要 · Abstract (English)
Properties of Fisher information matrices of 2-layer neural ReLU networks with random hidden weights are studied. For these networks, it is known that the eigenvalue distribution highly concentrates on several eigenspaces approximately. In particular, the eigenvalues for the first three eigenspaces account for 97.7% of the trace of the Fisher information matrix, independently of the number of parameters. In this paper, we identify the function spaces which correspond to those major eigenspaces. This function space consists of the spherical harmonic functions whose orders are not greater than 2. This result relates to the Mercer decomposition of the neural tangent kernels.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。