arXiv:2604.23720cs.LG2026-04中稿 · ICLR

提出准等变元网络,平衡对称性与表达力,提升权重空间学习效果。

Quasi-Equivariant Metanetworks

论文配图:Quasi-Equivariant Metanetworks
图 1 · 摘自论文原文
  • 引入准等变性概念,允许元网络在保持函数等价性前提下灵活设计
  • 在前馈、卷积和Transformer网络上验证,兼顾对称性与表达能力
  • 适合研究模型权重空间或设计高效元网络的研究者

元网络是直接作用于预训练权重以执行下游任务的神经架构。然而,参数空间仅是底层函数类的代理,参数-函数映射本质上非单射:不同的参数配置可能产生相同的输入输出行为。因此,仅依赖原始参数的元网络可能忽略架构的内在对称性。合理推理函数等价性对于有效元网络设计至关重要,这促使了等变元网络的发展,其通过引入等变性原则尊重架构对称性。但现有方法通常强制严格等变,导致约束过强,模型稀疏且表达力弱。为此,本文提出全新的准等变概念,使元网络在超越严格等变刚性的同时仍保持函数等价性。我们建立了该框架的理论基础,并证明其在前馈、卷积及Transformer等多种神经网络上的广泛适用性。实证结果表明,准等变元网络在对称性保留与表征表达力之间取得良好权衡。这些发现推进了权重空间学习的理论理解,为更富表达力和功能鲁棒的元网络设计提供了原则性基础。

原文摘要 · Abstract (English)

Metanetworks are neural architectures designed to operate directly on pretrained weights to perform downstream tasks. However, the parameter space serves only as a proxy for the underlying function class, and the parameter-function mapping is inherently non-injective: distinct parameter configurations may yield identical input-output behaviors. As a result, metanetworks that rely solely on raw parameters risk overlooking the intrinsic symmetries of the architecture. Reasoning about functional identity is therefore essential for effective metanetwork design, motivating the development of equivariant metanetworks, which incorporate equivariance principles to respect architectural symmetries. Existing approaches, however, typically enforce strict equivariance, which imposes rigid constraints and often leads to sparse and less expressive models. To address this limitation, we introduce the novel concept of quasi-equivariance, which allows metanetworks to move beyond the rigidity of strict equivariance while still preserving functional identity. We lay down a principled basis for this framework and demonstrate its broad applicability across diverse neural architectures, including feedforward, convolutional, and transformer networks. Through empirical evaluation, we show that quasi-equivariant metanetworks achieve good trade-offs between symmetry preservation and representational expressivity. These findings advance the theoretical understanding of weight-space learning and provide a principled foundation for the design of more expressive and functionally robust metanetworks.

元网络等变性权重空间函数等价

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。