arXiv:2608.12842cs.AI2026-08

提出CABS+模型融合方法,高效提升多任务模型性能。

CABS+: Efficient and Scalable Model Merging via Conflict-Aware Sparsification and Adaptive Weight Allocation

论文配图:CABS+: Efficient and Scalable Model Merging via Conflict-Aware Sparsification and Adaptive Weight Allocation
图 1 · 摘自论文原文
  • 通过自适应权重分配与冲突感知稀疏化,降低参数冲突影响。
  • 相比现有方法,整体性能提升16.97%,合并速度提升近4倍。
  • 适用于大中小模型及多种任务场景,资源消耗更低。

模型融合作为构建统一多任务模型的新兴范式,无需额外训练即可整合多个模型的知识。然而,任务间参数冲突与知识干扰常导致融合后性能下降。已有方法如CABS通过结构化剪枝和序列掩码缓解此问题,但依赖网格搜索确定缩放系数,导致时间复杂度呈指数增长,且优化目标易被高性能任务主导,影响整体效果。为此,本文提出CABS+:采用无梯度搜索的自适应权重分配(AWA)降低计算开销,设计非对称适应度函数以促进各任务均衡提升。此外,提出相对协同得分(RSS)量化模型可融合性,指导模型选择。在27个数据集、5种模型(涵盖大语言、小规模语言与视觉模型)上对比CABS、AdaMerging、WUDIMerging等先进方法,实验表明:相较AdaMerging和WUDIMerging,CABS+整体性能分别提升16.97%和12.93%,稳定性更强,跨任务数量与架构变化更具鲁棒性;所需GPU内存不足AdaMerging的25%,合并时间接近WUDIMerging的四分之一。

原文摘要 · Abstract (English)

Model merging has recently attracted significant attention as a promising paradigm for constructing unified multi-task models without requiring additional retraining. However, parameter conflicts and knowledge interference across tasks often degrade merged-model performance. Prior work introduced Conflict-Aware and Balanced Sparsification (CABS), which reduces parameter interference through structured pruning and sequential masking. However, CABS relies on grid search to determine scaling coefficients, resulting in exponential time complexity, while its optimization objective can be dominated by high-performance tasks, leading to suboptimal overall performance. To address these limitations, we extend CABS and propose CABS+. Specifically, Adaptive Weight Allocation (AWA) optimizes merging coefficients via a gradient-free search scheme to reduce time complexity, while an asymmetric fitness function promotes more comprehensive performance gains across tasks. Moreover, we conduct a systematic empirical study of key factors influencing model merging performance and propose Relative Synergy Score (RSS) to quantify model mergeability and guide model selection. We compare CABS+ with state-of-the-art model merging methods, including CABS, AdaMerging, and WUDIMerging, across 27 datasets and 5 models covering large language, small-scale language, and vision models. Extensive experiments verify the effectiveness and efficiency of CABS+. Compared with AdaMerging and WUDIMerging, CABS+ improves overall performance by 16.97% and 12.93%, respectively, exhibits stronger stability and robustness across varying task numbers and model architectures, uses less than 25% of the GPU memory required by AdaMerging, and achieves nearly a 4x speedup in merging time over WUDIMerging.

模型融合高效算法多任务学习稀疏化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。