通过部分融合神经网络,在模型性能与计算成本间实现灵活权衡。
Partial Fusion of Neural Networks: Efficient Tradeoffs Between Ensembles and Weight Aggregation

- 基于神经元相似性,仅融合最相似的神经元权重
- 使用部分最优传输方法匹配并聚合高相似度神经元
- 适用于需要平衡推理速度与精度的部署场景
神经网络集成通常比单个网络表现更好,但计算开销大;权重平均则更轻量,但准确率较低。本文提出神经网络的部分融合方法,介于集成与权重平均之间,可在计算成本与性能间灵活权衡。一种直接方式是基于神经元层面的相似性扩展现有权重平均方法,仅对最相似的神经元进行权重聚合。文中展示了一种结合神经元相似性识别与匹配的局部最优传输方法。此外,从广义剪枝视角看,部分融合可视为对集成模型的推广,允许神经元不仅被删除,还可线性组合。进一步发现,将该剪枝思想应用于单个网络,同样能实现隔离、删除或线性组合神经元的灵活权衡,效果接近部分融合。代码已开源。
原文摘要 · Abstract (English)
Ensembles of neural networks typically outperform individual networks but incur large computational costs, whereas weight aggregation produces less costly, yet also less accurate, aggregate models. We introduce partial fusion of networks, which interpolates between ensembles and weight aggregation and thus allows for a flexible tradeoff between computational cost and performance. A direct way to achieve this is to extend existing weight aggregation methods based on neuron-level similarity between different networks, where partial fusion then only aggregates weights of neurons which are most similar. We showcase one particular method to jointly identify which neurons are most similar and match them via partial optimal transport. Further, we consider the more general perspective of weight aggregation and partial fusion as generalized pruning of ensemble models, where neurons cannot just be deleted, but also linearly combined. Finally, we show that generalized pruning applied to a single network yields similar benefits as partial fusion by allowing for a tradeoff between isolating, deleting, and linearly combining neurons based on similarity. Our code is available at https://github.com/Fabian-Mor/partial_fusion_nn.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。