通过校准特征幅度提升模型融合性能,无需额外训练。
MAGIC: Achieving Superior Model Merging via Magnitude Calibration
- 提出幅度校准框架,修复融合时的特征幅度偏差
- 在8个视觉数据集上提升4.3%,Llama模型上提升8.0%
- 可插拔使用,支持无数据或少量数据场景
预训练模型的泛化催生了大量专用微调模型。模型融合旨在将这些专用模型的能力整合为统一模型,且几乎无需额外训练。核心目标是保持融合后模型的行为与原始模型一致,通常通过特征对齐实现。我们发现特征包含方向和幅度两个关键成分,现有研究多关注方向对齐,而忽略幅度受融合操作(如参数融合、稀疏化)扰动的影响。此类扰动导致融合模型特征偏离原始模型,引发性能下降。为此,我们提出可插拔的幅度校准框架MAGIC,包含三种变体:特征空间校准(FSC)利用少量无标签数据重对齐特征;权重空间校准(WSC)无需额外数据扩展至权重空间;二者结合形成双空间校准(DSC)。全面实验表明,MAGIC在多种计算机视觉任务(+4.3%在8个数据集)和NLP任务(+8.0%在Llama)中持续提升性能,且无需额外训练。代码已公开:https://github.com/lyymuwu/MAGIC
原文摘要 · Abstract (English)
The proliferation of pre-trained models has given rise to a wide array of specialised, fine-tuned models. Model merging aims to merge the distinct capabilities of these specialised models into a unified model, requiring minimal or even no additional training. A core objective of model merging is to ensure the merged model retains the behavioural characteristics of the specialised models, typically achieved through feature alignment. We identify that features consist of two critical components: direction and magnitude. Prior research has predominantly focused on directional alignment, while the influence of magnitude remains largely neglected, despite its pronounced vulnerability to perturbations introduced by common merging operations (e.g., parameter fusion and sparsification). Such perturbations to magnitude inevitably lead to feature deviations in the merged model from the specialised models, resulting in subsequent performance degradation. To address this, we propose MAGnItude Calibration (MAGIC), a plug-and-play framework that rectifies layer-wise magnitudes in feature and weight spaces, with three variants. Specifically, our Feature Space Calibration (FSC) realigns the merged model's features using a small set of unlabelled data, while Weight Space Calibration (WSC) extends this calibration to the weight space without requiring additional data. Combining these yields Dual Space Calibration (DSC). Comprehensive experiments demonstrate that MAGIC consistently boosts performance across diverse Computer Vision tasks (+4.3% on eight datasets) and NLP tasks (+8.0% on Llama) without additional training. Our code is available at: https://github.com/lyymuwu/MAGIC
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。