不依赖训练数据，用权重融合多模型，提升泛化能力。
On Defining Neural Averaging
- 将模型差异视为伪梯度，通过元优化融合权重
- 在分布外场景下性能优于单模型与模型汤方法
- 为无数据条件下模型聚合提供可扩展的统一框架
如何对多个预训练神经网络进行平均？本文研究从一组在互斥数据碎片上训练的模型中合成一个单一网络，仅使用其最终权重而无需访问原始训练数据。受模型汤（model soup）启发，该方法将模型平均重新诠释为更广义框架——无数据元集成（Amortized Model Ensembling, AME），将模型差异视为伪梯度，指导权重更新。此视角不仅复现了模型汤，还支持更灵活、自适应的集成策略。实验表明，AME生成的平均模型在分布外场景下显著优于个体专家与模型汤基线，验证了无数据权重聚合的原理性与通用性，定义了一种可行的神经网络平均范式。
原文摘要 · Abstract (English)
What does it even mean to average neural networks? We investigate the problem of synthesizing a single neural network from a collection of pretrained models, each trained on disjoint data shards, using only their final weights and no access to training data. In forming a definition of neural averaging, we take insight from model soup, which appears to aggregate multiple models into a singular model while enhancing generalization performance. In this work, we reinterpret model souping as a special case of a broader framework: Amortized Model Ensembling (AME) for neural averaging, a data-free meta-optimization approach that treats model differences as pseudogradients to guide neural weight updates. We show that this perspective not only recovers model soup but enables more expressive and adaptive ensembling strategies. Empirically, AME produces averaged neural solutions that outperform both individual experts and model soup baselines, especially in out-of-distribution settings. Our results suggest a principled and generalizable notion of data-free model weight aggregation and defines, in one sense, how to perform neural averaging.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。