通过神经元对齐融合多个模型,提升零样本和非独立同分布场景下的性能。
Model Fusion via Retrofitting
- 以中间神经元为单位,将多模型特征映射到统一表示空间。
- 在零样本和非独立同分布数据上,性能显著优于现有方法。
- 适用于多种架构,可处理任意模块化结构的深度网络。
模型融合旨在不重新训练的情况下合并多个独立训练的神经网络,但受排列不变性、随机初始化及异质训练数据引起的表征差异影响,实现困难。现有方法在非独立同分布数据下的零样本场景表现不佳,且常受限于特定架构或仅支持成对融合。本文提出一种以神经元为中心的融合算法族,将融合视为一个有原则的表征匹配问题:将父模型中的中间神经元分组为目标表征,再训练融合模型中对应的子网络去逼近这些表征。与以往工作不同,该方法引入神经元重要性得分,引导对关键特征的对齐,并可应用于任何可分解为层级有向无环图(DAG)的架构。在 VGG、ResNet 与 ViT 上的实证验证表明,该方法在标准基准测试中持续优于现有方法,尤其在零样本和非独立同分布场景下提升显著。代码已开源。
原文摘要 · Abstract (English)
Model fusion seeks to combine independently trained neural networks into a single model without retraining, but is complicated by representational divergence arising from permutation invariance, random initialization, and heterogeneous training data. Existing methods struggle particularly in zero-shot settings under non-IID data distributions, and are often limited to specific architectures or pairwise fusion. We introduce a neuron-centric family of fusion algorithms that frames fusion as a principled representation-matching problem: intermediate neurons across parent models are grouped into target representations, which the fused model's corresponding sub-networks are then trained to approximate. Unlike prior work, our approach incorporates neuron attribution scores to bias alignment toward salient features, and can be applied to any architecture modularizable as a DAG of levels -- empirically validated on VGGs, ResNets, and ViTs. Experiments across standard benchmarks show consistent improvements over existing fusion methods, with the largest gains in zero-shot and non-IID scenarios. Code is available at https://github.com/AndrewSpano/model-fusion-via-retrofitting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。