arXiv:2602.06552cs.LG2026-02被引 1

通过模块化专家重组,实现细粒度模型合并与高效推理。

Fine-Grained Model Merging via Modular Expert Recombination

  • 按组件分步合并模型,支持输入感知的动态模块重组。
  • 在多种任务和模型规模下优于基线,性能提升显著。
  • 适合需要高效推理与存储受限场景的应用开发者。

模型合并通过整合特定任务模型构建通用模型,无需标注数据或昂贵的联合训练。尽管近期方法通过为每个实例生成定制化合并模型提升了对异构任务的适应性,但仍面临两大挑战:一是实例专属合并模型缺乏复用性,限制了优质合并配置的利用和高效批量推理;二是将每个任务模型视为整体,忽视了同源组件(如注意力层、多层感知机)间不同的可合并性及组件间的合并敏感差异。为此,我们提出MERGE(模块化专家重组用于细粒度合并),支持组件级模型合并与推理时的输入感知、按需模块重组。MERGE将组件级合并建模为兼顾跨任务性能与存储效率的双目标优化问题,并采用代理辅助的进化算法高效搜索帕累托最优合并配置。这些高质量配置构成可复用的模块化专家库,由轻量路由网络动态激活并重组,以构建输入相关模型,在存储约束下实现高效推理。大量实验在不同模型规模、任务类型和微调策略下表明,MERGE持续优于强基线,且具有良好泛化能力。

原文摘要 · Abstract (English)

Model merging constructs versatile models by integrating task-specific models without requiring labeled data or expensive joint retraining. Although recent methods improve adaptability to heterogeneous tasks by generating customized merged models for each instance, they face two critical limitations. First, the instance-specific merged models lack reusability, restricting the exploitation of high-quality merging configurations and efficient batch inference. Second, these methods treat each task-specific model as a monolithic whole, overlooking the diverse mergeability of homologous components such as attention and multilayer perceptron layers, and the differing merging sensitivities across components. To address these limitations, we propose MERGE (\underline{M}odular \underline{E}xpert \underline{R}ecombination for fine-\underline{G}rained m\underline{E}rging), a method that enables component-wise model merging and input-aware, on-demand module recombination at inference. MERGE formulates component-wise merging as a bi-objective optimization problem that balances cross-task performance and storage efficiency, and develops a surrogate-assisted evolutionary algorithm to efficiently identify Pareto-optimal merging configurations. These high-quality configurations underpin a reusable modular expert library, from which a lightweight routing network dynamically activates and recombines modular experts to assemble input-specific models and enable efficient inference under storage constraints. Extensive experiments across various model scales, task types, and fine-tuning strategies demonstrate that MERGE consistently outperforms strong baselines and generalizes effectively.

模型合并模块化高效推理专家重组

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。