提出新型多项式等变网络,兼顾高效与强表达能力。
Equivariant Polynomial Functional Networks
- 基于参数共享构建多项式等变层,突破线性限制
- 在保持低内存和快速度的同时提升模型表达力
- 适合需要高效权重编辑与策略评估的场景
神经函数网络(NFNs)因其在隐式数据表示信息提取、网络权重编辑和策略评估等任务中的广泛应用而受到关注。现有NFNs通过图消息传递或参数共享机制实现排列与缩放等变性。但图基方法内存开销大、运行慢;参数共享方法虽高效,却受限于输入网络对称群规模,表达力不足。本文提出MAGEP-NFN(单项式矩阵群等变多项式神经函数网络),采用参数共享机制,以输入权重的多项式形式构建非线性等变层,引入不同隐藏层权重间的额外关系,显著增强表达力,同时保持低内存占用和快速推理。实验表明,MAGEP-NFN在性能与效率上均优于现有基线。
原文摘要 · Abstract (English)
Neural Functional Networks (NFNs) have gained increasing interest due to their wide range of applications, including extracting information from implicit representations of data, editing network weights, and evaluating policies. A key design principle of NFNs is their adherence to the permutation and scaling symmetries inherent in the connectionist structure of the input neural networks. Recent NFNs have been proposed with permutation and scaling equivariance based on either graph-based message-passing mechanisms or parameter-sharing mechanisms. However, graph-based equivariant NFNs suffer from high memory consumption and long running times. On the other hand, parameter-sharing-based NFNs built upon equivariant linear layers exhibit lower memory consumption and faster running time, yet their expressivity is limited due to the large size of the symmetric group of the input neural networks. The challenge of designing a permutation and scaling equivariant NFN that maintains low memory consumption and running time while preserving expressivity remains unresolved. In this paper, we propose a novel solution with the development of MAGEP-NFN (Monomial mAtrix Group Equivariant Polynomial NFN). Our approach follows the parameter-sharing mechanism but differs from previous works by constructing a nonlinear equivariant layer represented as a polynomial in the input weights. This polynomial formulation enables us to incorporate additional relationships between weights from different input hidden layers, enhancing the model's expressivity while keeping memory consumption and running time low, thereby addressing the aforementioned challenge. We provide empirical evidence demonstrating that MAGEP-NFN achieves competitive performance and efficiency compared to existing baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。