针对模型融合中的参数冗余问题,提出自适应稀疏化方法,提升融合效果。
One Size Does Not Fit All: A Distribution-Aware Sparsification for More Precise Model Merging
- 按参数张量分布特性动态分配稀疏度,避免统一比例导致误剪关键参数
- 在8个ViT-B/32任务上平均提升2.0%性能,显著优于传统稀疏策略
- 可无缝接入各类主流融合方法,适用于视觉、语言、多模态等多种场景
模型融合作为一种无需数据的多任务学习范式,能够将多个微调模型融合为一个强大单一模型。其中稀疏化是核心技巧,通过剪枝任务向量中的冗余参数来缓解参数干扰。然而,现有方法普遍采用‘一刀切’的统一稀疏率,忽略了参数在结构和统计上的异质性,常导致重要参数被误剪而无用参数保留。为此,我们提出TADrop(张量级自适应剪枝)方法,根据每个参数张量的分布特性动态分配稀疏水平:分布密集的张量可大幅剪枝,分布稀疏的则予以保留。作为轻量级可插拔模块,TADrop集成于基础、经典及前沿融合方法中。在视觉、语言与多模态多种任务和模型(ViT、BEiT)上的实验表明,其性能持续显著提升。例如,在领先融合方法基础上,8个ViT-B/32任务上平均提升2.0%。TADrop通过匹配模型结构实现更精准的干扰抑制,为高性能模型融合提供新基准。
原文摘要 · Abstract (English)
Model merging has emerged as a compelling data-free paradigm for multi-task learning, enabling the fusion of multiple fine-tuned models into a single, powerful entity. A key technique in merging methods is sparsification, which prunes redundant parameters from task vectors to mitigate interference. However, prevailing approaches employ a ``one-size-fits-all'' strategy, applying a uniform sparsity ratio that overlooks the inherent structural and statistical heterogeneity of model parameters. This often leads to a suboptimal trade-off, where critical parameters are inadvertently pruned while less useful ones are retained. To address this limitation, we introduce \textbf{TADrop} (\textbf{T}ensor-wise \textbf{A}daptive \textbf{Drop}), an adaptive sparsification strategy that respects this heterogeneity. Instead of a global ratio, TADrop assigns a tailored sparsity level to each parameter tensor based on its distributional properties. The core intuition is that tensors with denser, more redundant distributions can be pruned aggressively, while sparser, more critical ones are preserved. As a simple and plug-and-play module, we validate TADrop by integrating it with foundational, classic, and SOTA merging methods. Extensive experiments across diverse tasks (vision, language, and multimodal) and models (ViT, BEiT) demonstrate that TADrop consistently and significantly boosts their performance. For instance, when enhancing a leading merging method, it achieves an average performance gain of 2.0\% across 8 ViT-B/32 tasks. TADrop provides a more effective way to mitigate parameter interference by tailoring sparsification to the model's structure, offering a new baseline for high-performance model merging.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。