提出分层蒸馏方法，显著提升小样本模型合并效果。
Scalable Model Merging with Progressive Layer-wise Distillation
- 采用逐层蒸馏策略，逐步融合多模型能力
- 在视觉与NLU任务中分别提升6.14%和6.61%
- 支持超100亿参数模型，适合大规模模型整合场景
模型合并能有效整合多个微调模型的能力，但在缺乏或仅有少量数据时，合并模型性能下降仍是挑战。本文首次通过理论证明，无数据依赖的合并算法在最坏情况下表现可能极差，强调领域特定数据的重要性。基于此，我们揭示了模型合并与知识蒸馏的关系，提出一种新的少样本合并算法ProDistill（渐进式分层蒸馏）。不同于普遍认为分层训练会损害性能的观点，我们发现分层教师-学生蒸馏不仅能提升可扩展性，还能增强合并效果。大量实验表明，相比现有少样本合并方法，ProDistill在视觉和自然语言理解任务上分别实现最高6.14%和6.61%的性能提升。此外，实验扩展至超过100亿参数的模型，验证了ProDistill出色的可扩展性。
原文摘要 · Abstract (English)
Model merging offers an effective way to integrate the capabilities of multiple fine-tuned models. However, the performance degradation of the merged model remains a challenge, particularly when none or few data are available. This paper first highlights the necessity of domain-specific data for model merging by proving that data-agnostic algorithms can have arbitrarily bad worst-case performance. Building on this theoretical insight, we explore the relationship between model merging and distillation, introducing a novel few-shot merging algorithm, ProDistill (Progressive Layer-wise Distillation). Unlike common belief that layer wise training hurts performance, we show that layer-wise teacher-student distillation not only enhances the scalability but also improves model merging performance. We conduct extensive experiments to show that compared to existing few-shot merging methods, ProDistill achieves state-of-the-art performance, with up to 6.14% and 6.61% improvements in vision and NLU tasks. Furthermore, we extend the experiments to models with over 10B parameters, showcasing the exceptional scalability of ProDistill.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。