无需训练即可生成任意尺寸的模型,灵活适配不同硬件环境。
Boomerang Distillation Enables Zero-Shot Model Size Interpolation
- 从大模型逐步蒸馏再重建,实现无训练的模型尺寸插值。
- 中间尺寸模型性能平滑过渡,常优于同尺寸预训练模型。
- 适合资源受限场景下的模型部署与快速适配。
大型语言模型在实际部署中面临多样化的内存与算力限制。现有方法需独立训练每个尺寸模型,成本高昂且仅支持粗粒度尺寸选择。本文提出一种新现象——回旋蒸馏:从大模型(教师)出发,先蒸馏至小模型(学生),再通过逐步重加入教师层来重建中间尺寸模型,无需额外训练。该过程生成零样本插值模型,其性能在学生与教师间平滑变化,通常优于或匹配同尺寸的预训练或蒸馏模型。我们进一步分析发现,教师与学生间通过剪枝和蒸馏实现对齐是成功的关键。回旋蒸馏提供了一种简单高效的方法,可大幅降低训练成本,实现细粒度模型家族构建,灵活适应各类部署环境。代码与模型见 https://github.com/dcml-lab/boomerang-distillation。
原文摘要 · Abstract (English)
Large language models (LLMs) are typically deployed under diverse memory and compute constraints. Existing approaches build model families by training each size independently, which is prohibitively expensive and provides only coarse-grained size options. In this work, we identify a novel phenomenon that we call boomerang distillation: starting from a large base model (the teacher), one first distills down to a small student and then progressively reconstructs intermediate-sized models by re-incorporating blocks of teacher layers into the student without any additional training. This process produces zero-shot interpolated models of many intermediate sizes whose performance scales smoothly between the student and teacher, often matching or surpassing pretrained or distilled models of the same size. We further analyze when this type of interpolation succeeds, showing that alignment between teacher and student through pruning and distillation is essential. Boomerang distillation thus provides a simple and efficient way to generate fine-grained model families, dramatically reducing training cost while enabling flexible adaptation across deployment environments. The code and models are available at https://github.com/dcml-lab/boomerang-distillation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。