提出CABS框架,解决模型合并中的参数冲突与分布不均问题。
CABS: Conflict-Aware and Balanced Sparsification for Enhancing Model Merging
- 通过顺序剪枝加掩码减少任务向量间参数重叠。
- 采用n:m剪枝策略保持关键权重并均衡各层分布。
- 在多种模型和任务上优于现有方法,适合多任务模型整合场景。
基于任务向量(即微调模型与共享基础模型的参数差异)的模型合并,可在不重新训练的情况下高效整合多个特定任务模型为多任务模型。近期工作尝试通过稀疏化缓解任务向量间的冲突,但受制于高参数重叠和权重分布不均两大问题。为此,本文提出简单而有效的CABS(Conflict-Aware and Balanced Sparsification)框架,包含冲突感知稀疏化(CA)和均衡稀疏化(BS)。CA通过在序列剪枝中应用掩码,降低参数重叠,确保每个任务向量保留独立且非重叠的参数;BS采用n:m剪枝策略,在保留关键权重的同时实现层间分布均衡。大量实验表明,CABS在多种任务和模型规模下均超越当前最优方法。
原文摘要 · Abstract (English)
Model merging based on task vectors, i.e., the parameter differences between fine-tuned models and a shared base model, provides an efficient way to integrate multiple task-specific models into a multitask model without retraining. Recent works have endeavored to address the conflicts between task vectors, one of the significant challenges faced by model merging, through sparsification; however, two issues significantly limit their performance: high parameter overlap and unbalanced weight distribution. To address these issues, we propose a simple, yet effective framework called CABS (Conflict-Aware and Balanced Sparsification), consisting of Conflict-Aware Sparsification (CA) and Balanced Sparsification (BS). CA can reduce parameter overlap by applying masks during sequential pruning, ensuring that each task vector retains distinct, non-overlapping parameters. BS leverages $n$: $m$ pruning to preserve critical weights while maintaining an even distribution across layers. Our comprehensive experiments demonstrate that CABS outperforms state-of-the-art methods across diverse tasks and model sizes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。