让不同架构的模型共用一个权重空间，提升知识迁移能力
The Impact of Model Zoo Size and Composition on Weight Space Learning
- 改造通用方法，支持异构模型群体训练
- 不同数据集训练的模型能显著提升生成效果
- 适合需要跨模型知识迁移的研究者
复用已训练的神经网络模型是降低训练成本、传递知识的常用策略。权重空间学习——将预训练模型的权重作为数据模态——是复用模型群体的新兴领域，在模型分析和权重生成任务中表现出色。然而，现有方法要求模型池中所有模型具有完全相同的架构，限制了其在更广泛模型上的泛化能力。本文突破此限制，提出对主流权重空间学习方法的改进，使其可处理异构模型群体。我们进一步研究模型多样性对零样本知识迁移中生成未见模型权重的影响。大量实验表明，包含不同图像数据集训练的模型能显著提升性能与泛化能力，无论是在分布内还是分布外设置下。代码已开源：github.com/HSG-AIML/MultiZoo-SANE。
原文摘要 · Abstract (English)
Re-using trained neural network models is a common strategy to reduce training cost and transfer knowledge. Weight space learning - using the weights of trained models as data modality - is a promising new field to re-use populations of pre-trained models for future tasks. Approaches in this field have demonstrated high performance both on model analysis and weight generation tasks. However, until now their learning setup requires homogeneous model zoos where all models share the same exact architecture, limiting their capability to generalize beyond the population of models they saw during training. In this work, we remove this constraint and propose a modification to a common weight space learning method to accommodate training on heterogeneous populations of models. We further investigate the resulting impact of model diversity on generating unseen neural network model weights for zero-shot knowledge transfer. Our extensive experimental evaluation shows that including models with varying underlying image datasets has a high impact on performance and generalization, for both in- and out-of-distribution settings. Code is available on github.com/HSG-AIML/MultiZoo-SANE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。