按参数重要性差异融合模型，提升合并效果
Non-Uniform Parameter-Wise Model Merging
- 基于梯度优化学习每个参数的贡献权重
- 在多种架构和场景下均优于传统合并方法
- 支持多模型融合，适合分布式模型集成
融合多个机器学习模型是提升性能的常用技术，尤其适用于分布式环境。传统方法如模型集成虽有效，但内存与计算开销大。近期基于参数平均的方法在某些场景表现良好，但对初始化不同且训练轨迹无重叠的模型，即使对齐神经元，合并后性能仍可能低于原始模型。本文提出非均匀参数级模型融合（NP Merge），通过梯度优化学习每个参数对最终模型的贡献。实验证明，该方法在多种架构和设置下均优于以往方法，并可扩展至多模型融合，具备良好的可扩展性与鲁棒性。
原文摘要 · Abstract (English)
Combining multiple machine learning models has long been a technique for enhancing performance, particularly in distributed settings. Traditional approaches, such as model ensembles, work well, but are expensive in terms of memory and compute. Recently, methods based on averaging model parameters have achieved good results in some settings and have gained popularity. However, merging models initialized differently that do not share a part of their training trajectories can yield worse results than simply using the base models, even after aligning their neurons. In this paper, we introduce a novel approach, Non-uniform Parameter-wise Model Merging, or NP Merge, which merges models by learning the contribution of each parameter to the final model using gradient-based optimization. We empirically demonstrate the effectiveness of our method for merging models of various architectures in multiple settings, outperforming past methods. We also extend NP Merge to handle the merging of multiple models, showcasing its scalability and robustness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。