通过层拼接实现模型大小插值,提升中间尺寸模型性能
Understanding Layer Patching in Model Size Interpolation

- 将层选择问题建模为最短路径优化,找到最优层组合方式
- 简单顺序拼接(从头或从尾)常表现优异,效果接近最优
- 提出KLPatch算法,基于KL散度贪心拼接,进一步提升性能
零样本模型大小插值旨在不经过额外训练的情况下,通过组合已有模型生成目标中间尺寸的新模型。近期研究发现,可通过迭代层拼接方式扩展学生模型:用教师模型中连续的层块替换学生模型的层,从而得到大小与性能介于师生之间的模型。本文首次系统研究学生模型层选择策略。我们将每种模型尺寸下的最优层子集选取建模为优化问题,并证明其可视为特定有向无环图中的最短路径问题。实验表明,层拼接显著影响插值行为,且在不同模型家族间差异明显。我们发现,简单的顺序拼接策略——从第一层到最后一层或反之——在实践中往往表现惊人良好。此外,我们提出KLPatch算法,一种基于KL散度的贪心拼接方法,通常优于最后到最先的拼接策略,并近似求解该优化问题。整体结果为层拼接如何影响模型大小插值提供了理论支撑,并为构建近优插值模型提供实用指导。
原文摘要 · Abstract (English)
Zero-shot model size interpolation aims to create new models of intermediate target sizes by combining existing models without additional training. Recent work on boomerang distillation [Kangaslahti et al., 2026] shows that a student language model distilled from a larger teacher can be expanded by iteratively patching its layers, replacing student layers with contiguous blocks of teacher layers to obtain models whose size and performance interpolate between the student and the teacher. In this work, we provide the first systematic study of student-layer selection for model size interpolation. We cast finding the optimal layer subset for each model size as an optimization problem and prove it can be viewed as a shortest-path problem in a certain acyclic graph. In experiments, we show that patching strongly shapes interpolation behavior, with effects that vary substantially across model families. We find that simple sequential strategies--patching either from the first layer to the last or from the last to the first--often achieve surprisingly strong performance in practice. We further introduce KLPatch, a greedy patching algorithm based on KL divergence, which often improves over last-to-first patching and approximately solves the optimization problem. Together, our results provide a principled understanding of how layer patching affects model size interpolation and offer practical guidance for constructing near-optimal interpolated models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。