让训练时的权重与部署时一致,提升压缩模型性能。
Train What You Deploy: Closing the MLP Reachability Gap in Low-Rank Clone Distillation
- 训练时直接优化部署用的完整权重矩阵,不局限于教师模型的子集。
- 在相同参数量下,相比基线提升2.36~10.45分,最高效率提升2倍。
- 适合追求高精度低资源压缩的模型部署研究者使用。
压缩后的学生模型在推理时使用的权重结构与其训练时能到达的权重范围可能不一致。我们发现,当前先进的低秩克隆(LRC)蒸馏方法虽部署全尺寸多层感知机(MLP),但训练时仅限于教师模型诱导的子空间,导致每个部署矩阵有62.5%-81.4%的独立线性自由度无法触及。提出核心原则:训练即部署。基于相同的LRC初始化,通过两种可合并实现(Dense-LRC和CORE-LRC),将训练目标扩展至整个部署权重矩阵,保持部署形状、参数量和推理浮点运算量不变。该方法恢复了被浪费的容量:在三个教师模型(Llama3.2-3B, Llama3.1-8B, Qwen2.5-3B)上,相较同预算的Plain-LRC基线,平均准确率分别提升+2.36/+2.71/+10.45,最大增益出现在最宽的教师模型(Qwen)上,仅用10B蒸馏令牌即达到原配方约20B令牌的性能(2倍令牌效率)。即使在同一谱系中,仍获得+6.39的提升。控制实验表明,性能提升源于可访问权重空间扩大,而非新增参数或训练方案。仅需约10B蒸馏令牌加短时间SFT,1.5B学生模型即可匹配约9T令牌教师的9任务宏平均表现,误差在评估噪声内,仅存在微小MMLU差距;2.7B学生模型在约900倍更少的压缩令牌下超越Meta官方对Llama3.1-8B的压缩结果(令牌数低于未匹配方案,非计算量宣称)。所有结果均来自单种子运行的LRC主干。
原文摘要 · Abstract (English)
A compressed student has two shapes that need not agree: the weight it deploys at inference and the weight family its training can reach. We show that a state-of-the-art weight-inheritance distiller, Low-Rank Clone (LRC), deploys a full-width student MLP but ties training to a teacher-induced slice, leaving 62.5-81.4% of each deployed matrix's independent linear degrees of freedom unreachable-paid for at inference, never trainable. Our principle is one line: train what you deploy. From the identical LRC warm start, we make the training object the entire deployed matrix, with no change in deployed shape, deployed parameter count, or inference FLOPs, via two mergeable realizations (Dense-LRC and CORE-LRC) that both collapse to one deployed weight. This recovers stranded capacity: taking the stronger realization per teacher, +2.36/+2.71/+10.45 Avg9 over matched-budget plain-LRC baselines across three teachers (Llama3.2-3B, Llama3.1-8B, Qwen2.5-3B), with the largest gain on the widest teacher (Qwen), where it reaches the original recipe's approx. 20B-token accuracy at 10B tokens (2x token efficiency); there the strictly same-lineage arm still recovers +6.39, the fully controlled figure. Controls strongly support attributing the gain to the enlarged reachable set, rather than to added parameters or the recipe. From approx. 10B distillation tokens plus a short SFT, a half-parameter 1.5B student matches its approx. 9T-token teacher's 9-task macro-average, within evaluation noise and with a residual MMLU deficit, and a 2.7B student beats Meta's own official compression of Llama3.1-8B at ~900x fewer compression tokens (a token count under unmatched recipes, not a compute claim). All results are from single-seed runs on the LRC backbone.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。