揭示浅层ReLU网络参数对称性的完整分类规律。
A Complete Symmetry Classification of Shallow ReLU Networks
- 利用ReLU非光滑性分析参数对称性
- 首次完成浅层ReLU网络对称性完整分类
- 适用于研究优化动力学与模型可解释性
神经网络的参数空间并非函数空间。这一现象早在1990年代就以‘逆向工程’或‘参数可辨识性’等形式被研究，进而引出参数空间对称性问题——即不同参数却实现相同函数的现象。通过将产生相同函数的参数进行等价类划分，得到的商空间称为‘神经流形’（neuromanifold），其几何结构已被证明会影响优化动力学。此前的完整分类方法依赖激活函数的解析性，排除了重要的ReLU情形。本文则反其道而行之，利用ReLU的非可微性，首次实现了浅层ReLU网络对称性的完整分类。
原文摘要 · Abstract (English)
Parameter space is not function space for neural network architectures. This fact, investigated as early as the 1990s under terms such as ``reverse engineering," or ``parameter identifiability", has led to the natural question of parameter space symmetries\textemdash the study of distinct parameters in neural architectures which realize the same function. Indeed, the quotient space obtained by identifying parameters giving rise to the same function, called the \textit{neuromanifold}, has been shown in some cases to have rich geometric properties, impacting optimization dynamics. Thus far, techniques towards complete classifications have required the analyticity of the activation function, notably excising the important case of ReLU. Here, in contrast, we exploit the non-differentiability of the ReLU activation to provide a complete classification of the symmetries in the shallow case.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。