提出新型神经网络架构,通过扩展对称性提升模型效率。
Monomial Matrix Group Equivariant Neural Functional Networks
- 引入单项矩阵群对称性,融合权重缩放与符号翻转
- 参数量显著减少,保持竞争力性能
- 适用于全连接与卷积网络,理论与实证兼备
神经函数网络(NFNs)因其在预测网络泛化、网络编辑及隐式神经表示分类等任务中的广泛应用而受到关注。现有NFN设计多依赖于隐藏层神经元无序排列带来的置换对称性,但未考虑ReLU网络的权重缩放对称性以及sin或Tanh网络的权重符号翻转对称性。本文将网络权重上的群作用从置换矩阵群扩展至单项矩阵群,通过设计相应的等变与不变层来编码缩放与符号翻转对称性。由此提出的新型NFN称为单项矩阵群等变神经函数网络(Monomial-NFN)。由于对称性扩展,Monomial-NFN相比文献中基线模型具有更少的独立可训练参数,显著提升模型效率。针对全连接与卷积神经网络,我们理论上证明了所有在权重空间上保持网络不变的群均为单项矩阵群的子群。实验验证了该模型在性能与效率上的优势。
原文摘要 · Abstract (English)
Neural functional networks (NFNs) have recently gained significant attention due to their diverse applications, ranging from predicting network generalization and network editing to classifying implicit neural representation. Previous NFN designs often depend on permutation symmetries in neural networks' weights, which traditionally arise from the unordered arrangement of neurons in hidden layers. However, these designs do not take into account the weight scaling symmetries of $\ReLU$ networks, and the weight sign flipping symmetries of $\sin$ or $\Tanh$ networks. In this paper, we extend the study of the group action on the network weights from the group of permutation matrices to the group of monomial matrices by incorporating scaling/sign-flipping symmetries. Particularly, we encode these scaling/sign-flipping symmetries by designing our corresponding equivariant and invariant layers. We name our new family of NFNs the Monomial Matrix Group Equivariant Neural Functional Networks (Monomial-NFN). Because of the expansion of the symmetries, Monomial-NFN has much fewer independent trainable parameters compared to the baseline NFNs in the literature, thus enhancing the model's efficiency. Moreover, for fully connected and convolutional neural networks, we theoretically prove that all groups that leave these networks invariant while acting on their weight spaces are some subgroups of the monomial matrix group. We provide empirical evidence to demonstrate the advantages of our model over existing baselines, achieving competitive performance and efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。