通过移植关键神经元融合模型，提升性能并减少资源占用。
Model Fusion via Neuron Transplantation
- 将重要神经元从多个模型移植到剪枝后空出的空间中
- 微调后性能超越同规模单模型，且训练推理更高效
- 适合需要轻量化高精度模型的部署场景
集成学习能有效提升神经网络预测性能，但带来内存和推理时间增加。本文提出一种新型模型融合方法——神经元移植（Neuron Transplantation, NT），通过将多个模型中的重要神经元移植到剪枝后留出的空隙中实现融合。移植后性能短暂下降可通过少量微调快速恢复，且始终优于相同容量与结构的单个模型。此外，NT支持所有集成模型联合剪枝与联合训练。相比基于对齐的平均方法（如最优传输融合），其融合过程更快、内存更低，所需微调更少，性能相当或更优。代码已开源。
原文摘要 · Abstract (English)
Ensemble learning is a widespread technique to improve the prediction performance of neural networks. However, it comes at the price of increased memory and inference time. In this work we propose a novel model fusion technique called \emph{Neuron Transplantation (NT)} in which we fuse an ensemble of models by transplanting important neurons from all ensemble members into the vacant space obtained by pruning insignificant neurons. An initial loss in performance post-transplantation can be quickly recovered via fine-tuning, consistently outperforming individual ensemble members of the same model capacity and architecture. Furthermore, NT enables all the ensemble members to be jointly pruned and jointly trained in a combined model. Comparing it to alignment-based averaging (like Optimal-Transport-fusion), it requires less fine-tuning than the corresponding OT-fused model, the fusion itself is faster and requires less memory, while the resulting model performance is comparable or better. The code is available under the following link: https://github.com/masterbaer/neuron-transplantation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。