通过参数归一化实现模型合并，让相似网络自然收敛到同一区域。
Symmetry-Aware Graph Metanetwork Autoencoders: Model Merging through Parameter Canonicalization
- 用尺度不变的图元网络做编码器，同时处理排列和缩放对称性
- 实验表明可无须求解组合分配问题，直接对齐INRs和CNNs
- 适合需要平滑模型融合的场景，如大模型集成与压缩
神经网络参数化在损失曲面上存在固有的对称性，导致多个等效极小值共存。尺度图元网络（ScaleGMNs）通过设计对排列和参数缩放变换都保持等变的架构，显式利用这些对称性。Ainsworth等人（2023）曾通过计算复杂的组合分配问题解决排列对称性，证明仅利用排列对称性即可将网络映射至共享损失盆地。本文进一步引入缩放对称性，提出一种基于ScaleGMNs的自编码器框架，作为不变编码器。实验表明，该方法可在不显式求解分配问题的前提下，同时在排列和缩放对称性下对齐隐式神经表示（INRs）与卷积神经网络（CNNs）。此策略确保相似网络自然收敛于同一盆地，从而实现平滑的模型合并——即线性插值过程避开高损失区域。代码已开源。
原文摘要 · Abstract (English)
Neural network parameterizations exhibit inherent symmetries that yield multiple equivalent minima within the loss landscape. Scale Graph Metanetworks (ScaleGMNs) explicitly leverage these symmetries by proposing an architecture equivariant to both permutation and parameter scaling transformations. Previous work by Ainsworth et al. (2023) addressed permutation symmetries through a computationally intensive combinatorial assignment problem, demonstrating that leveraging permutation symmetries alone can map networks into a shared loss basin. In this work, we extend their approach by also incorporating scaling symmetries, presenting an autoencoder framework utilizing ScaleGMNs as invariant encoders. Experimental results demonstrate that our method aligns Implicit Neural Representations (INRs) and Convolutional Neural Networks (CNNs) under both permutation and scaling symmetries without explicitly solving the assignment problem. This approach ensures that similar networks naturally converge within the same basin, facilitating model merging, i.e., smooth linear interpolation while avoiding regions of high loss. The code is publicly available on our GitHub repository.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。