arXiv:2506.18135cs.AIcs.CL2025-06中稿 · IJCNN2025被引 2

动态融合多任务模型,自适应增强专家能力。

SE-Merging: A Self-Enhanced Approach for Dynamic Model Merging

  • 基于表征分析,动态识别输入任务并调整融合系数。
  • 无需额外训练,在多个数据集上提升多任务性能。
  • 适合需要高效多任务部署的场景,如边缘计算。

模型融合因其能通过插值不同任务微调模型的参数实现多任务能力而受到关注。然而,其内在机制仍不清晰。本文从表征视角深入分析发现,模型融合成功的关键在于两点:一是区分来自不同任务的样本,二是为每个样本适配对应的专家模型。基于此,我们提出 exttt{SE-Merging} 框架,利用这两项特性动态识别输入样本的任务,并自适应地缩放融合系数,以进一步增强合并模型中的任务专长。值得注意的是, exttt{SE-Merging} 在无需额外训练的情况下实现了动态融合。大量实验表明,该方法在保持与现有融合技术兼容的同时,显著提升了性能。

原文摘要 · Abstract (English)

Model merging has gained increasing attention due to its intriguing property: interpolating the parameters of different task-specific fine-tuned models leads to multi-task abilities. However, despite its empirical success, the underlying mechanisms of model merging remain poorly understood. In this work, we delve into the mechanism behind model merging from a representation perspective. Our analysis reveals that model merging achieves multi-task abilities through two key capabilities: i) distinguishing samples from different tasks, and ii) adapting to the corresponding expert model for each sample. These two capabilities allow the merged model to retain task-specific expertise, enabling efficient multi-task adaptation. Building on these insights, we propose \texttt{SE-Merging}, a self-enhanced model merging framework that leverages these two characteristics to dynamically identify the corresponding task for each sample and then adaptively rescales the merging coefficients to further enhance task-specific expertise in the merged model. Notably, \texttt{SE-Merging} achieves dynamic model merging without additional training. Extensive experiments demonstrate that \texttt{SE-Merging} achieves significant performance improvements while remaining compatible with existing model merging techniques.

模型融合多任务学习自适应

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。