arXiv:2508.21421cs.LG2025-08被引 3

提出链式合并方法,解决多模型融合时层间依赖问题。

Rethinking Layer-wise Model Merging through Chain of Merges

  • 逐层合并权重并同步更新激活统计量
  • 在多个基准上达到当前最优性能
  • 适合需要融合多个微调模型的场景

微调预训练模型已成为众多领域实现顶尖性能的标准路径,导致大量针对特定任务的模型变体涌现。随着这类专用模型数量增加,不经过重新训练将其合并为统一模型成为关键挑战。现有合并技术仅在单个层级别操作,忽略了深度网络中固有的层间依赖关系。我们发现这种简化会导致分布偏差,尤其在依赖中间激活的方法中,早期层的变化无法有效传递至下游层。我们将此现象识别为内部协变量偏移,类似于神经网络训练初期的类似问题。为此,我们提出链式合并(Chain of Merges, CoM),一种逐层合并权重并同步更新激活统计量的层级合并方法。通过显式考虑层间交互,CoM 缓解了协变量偏移,通过一系列条件最优更新生成连贯的合并模型。在标准基准上的实验表明,CoM 实现了当前最优性能。

原文摘要 · Abstract (English)

Fine-tuning pretrained models has become a standard pathway to achieve state-of-the-art performance across a wide range of domains, leading to a proliferation of task-specific model variants. As the number of such specialized models increases, merging them into a unified model without retraining has become a critical challenge. Existing merging techniques operate at the level of individual layers, thereby overlooking the inter-layer dependencies inherent in deep networks. We show that this simplification leads to distributional mismatches, particularly in methods that rely on intermediate activations, as changes in early layers are not properly propagated to downstream layers during merging. We identify these mismatches as a form of internal covariate shift, comparable to the phenomenon encountered in the initial phases of neural networks training. To address this, we propose Chain of Merges (CoM), a layer-wise merging procedure that sequentially merges weights across layers while sequentially updating activation statistics. By explicitly accounting for inter-layer interactions, CoM mitigates covariate shift and produces a coherent merged model through a series of conditionally optimal updates. Experiments on standard benchmarks demonstrate that CoM achieves state-of-the-art performance.

模型融合层间依赖协同训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。