提出新方法提升模型合并的泛化能力,避免依赖经验调参。
Model Merging via Multi-Teacher Knowledge Distillation
- 将模型合并建模为多教师知识蒸馏,利用无标签数据优化参数融合。
- 理论证明最小化教师-学生KL散度可收紧合并模型的误差上界。
- 基于平坦极小值搜索,实现跨视觉与NLP任务的性能新纪录。
模型合并作为多任务学习的轻量级替代方案,其泛化能力尚未被充分理解。合并过程通常无法访问原始训练数据,且需融合在显著异构数据分布上微调的模型。缺乏对这一动态的原理性认识,导致现有方法依赖启发式策略确定各模型贡献权重(系数缩放),易受初始化影响且性能不稳定。本文首先建立针对模型合并场景的新型平坦性感知的PAC-Bayes泛化界,引入“跨任务异质性”项以形式化刻画不同微调模型先验与目标多任务分布之间的偏差。基于此理论洞察,将模型合并重构为在稀缺无标签数据上的多教师知识蒸馏问题,并严格证明最小化学生-教师KL散度能直接收紧合并模型的超额风险上界。进一步提出SAMerging方法,通过锐度感知最小化(SAM)寻找平坦极小值以实现该目标。实验表明,SAMerging在多个视觉与自然语言处理基准上均达到新最优性能。代码已开源。
原文摘要 · Abstract (English)
Model merging has emerged as a lightweight alternative to joint multi-task learning (MTL), yet the generalization properties of merged models remain largely unexplored. Establishing such theoretical guarantees is non-trivial, as the merging process typically forbids access to the original training data and involves combining fine-tuned models trained on fundamentally heterogeneous data distributions. Without a principled understanding of these dynamics, current methods often rely on heuristics to approximate the optimal combination of parameters. This dependence is most critical in coefficient scaling, the weighting factors that modulate the magnitude of each fine-tuned model's contribution to the shared parameter. However, without a principled objective to guide their selection, these methods lead to brittle performance and are highly sensitive to scaling initialization. We address this gap by (i) establishing a novel flatness-aware PAC-Bayes generalization bound specifically for the model merging setting. This analysis introduces a "cross-task heterogeneity" term that formally captures the mismatch between diverse fine-tuned model priors and the target multi-task distributions. Guided by this theoretical insight, (ii) we frame model merging as multi-teacher knowledge distillation on scarce, unlabeled data. We formally demonstrate that minimizing the student-teacher Kullback-Leibler divergence directly tightens the upper bound on the merged model's excess risk. Guided by the flatness-aware bound derived, (iii) we operationalize this objective via SAMerging, a method that employs Sharpness-Aware Minimization (SAM) to find flat minima. Empirically, SAMerging establishes a new state of the art across vision and NLP benchmarks, achieving remarkable performance. The code is available at https://github.com/arshandalili/SAMerging.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。