arXiv:2509.25712cs.LG2025-09被引 4

无需标注数据,自动优化多专家模型融合参数,提升性能且更高效。

Expert Merging: Model Merging with Unsupervised Expert Alignment and Importance-Guided Layer Chunking

  • 用无标签数据学习分层系数,对齐专家模型的隐藏状态和输出结果。
  • 在多个主流模型上超越现有训练免费与训练依赖方法,部分场景优于有监督训练。
  • 按层重要性动态分配参数,高价值层更精细调整,低价值层保持轻量。

模型融合将多个领域专精的专家模型合并为单一模型,可在不进行联合训练或部署多个模型的前提下,为大语言模型(LLMs)和多模态大语言模型(MLLMs)赋予广泛能力。然而,免训练方法依赖人工调参,而训练型方法主要对齐参数而非下游任务行为,且通常对所有层一视同仁,忽视层间差异。我们提出 Expert Merging,一种轻量级训练方法,仅使用无标签校准数据学习一组分层系数。这些系数通过优化使融合模型的隐状态和输出概率与对应专家模型对齐,并引入正则项保证稳定性、任务加权损失实现可控权衡。为捕捉层间差异,Expert Merging++ 进一步引入重要性引导分块机制:基于学习到的系数、任务向量幅度和参数数量构建归一化层重要性度量,将更多分块系数分配给高重要性层,同时保持低重要性层轻量化。该方法实现了无需标签、参数高效且可扩展的多专家模型融合,在 InternVL、Qwen2-VL 和 Mistral 等 MLLM 与 LLM 骨干网络上均超越强基准方法,Expert Merging++ 更进一步提升性能,某些情况下甚至超过有监督混合训练。代码已开源。

原文摘要 · Abstract (English)

Model merging, which combines multiple domain-specialized experts into a single model, offers a practical path to endow Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) with broad capabilities without the cost of joint training or serving many models. However, training-free methods rely on hand-tuned coefficients, whereas training-based methods primarily align parameters rather than downstream task behavior and typically treat all layers uniformly, ignoring inter-layer heterogeneity. We introduce Expert Merging, a training-light method that learns a small set of layer-wise coefficients using only unlabeled calibration data. The coefficients are optimized to explicitly align the merged model's hidden states and logits with those of the corresponding experts, with a coefficient regularizer for stability and task-weighted losses for controllable trade-offs. To capture inter-layer variation, Expert Merging++ augments this design with importance-guided chunking: a normalized layer-importance metric, derived from learned coefficients, task-vector magnitudes, and parameter counts, allocates more chunk-wise coefficients to high-importance layers while keeping low-importance layers lightweight. The result is a label-free, parameter-efficient, and scalable approach to multi-expert model merging across LLMs and MLLMs. Across MLLM backbones (InternVL and Qwen2-VL) and the LLM backbone (Mistral), our method surpasses strong training-free and training-based merging baselines, with Expert Merging++ delivering further gains and, in some cases, even exceeding supervised Mixture Training. The source code is available at https://github.com/Littleor/ExpertMerging.

模型融合大模型参数效率多专家

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。