通过参数空间对齐，让小模型更接近大模型的内部表示，提升性能且无额外开销。
GUIDE: Guided Initialization and Distillation of Embeddings
- 让小模型在参数空间上匹配大模型，而不仅是输出结果
- 400M-1B参数模型在200亿词上训练时，师生模型差距缩小25%-26%
- 可与传统知识蒸馏结合使用，效果近乎叠加，且不增加计算成本
如知识蒸馏等算法效率技术可在不增加推理成本的前提下提升模型质量，前提是存在一个更大的教师模型供小型学生模型学习。标准蒸馏方法仅要求学生模型匹配教师模型的输出，但训练大型模型成本高昂，因此我们应从教师模型中提取更多有用信息。本文提出 extit{GUIDE}（Guided Initialization and Distillation of Embeddings），一种迫使学生模型在参数空间上匹配教师模型的蒸馏方法。实验表明，在约200亿词的数据上，使用400M至10亿参数的学生模型时，师生模型间的质量差距降低了25%-26%。我们还进行了详尽分析，证明GUIDE可与知识蒸馏结合，实现近乎加成的效果。更重要的是，该方法不引入任何训练或推理开销，所有性能提升近乎免费。
原文摘要 · Abstract (English)
Algorithmic efficiency techniques such as distillation (\cite{hinton2015distillation}) are useful in improving model quality without increasing serving costs, provided a larger teacher model is available for a smaller student model to learn from during training. Standard distillation methods are limited to only forcing the student to match the teacher's outputs. Given the costs associated with training a large model, we believe we should be extracting more useful information from a teacher model than by just making the student match the teacher's outputs. In this paper, we introduce \guide (Guided Initialization and Distillation of Embeddings). \guide can be considered a distillation technique that forces the student to match the teacher in the parameter space. Using \guide we show 25-26\% reduction in the teacher-student quality gap when using large student models (400M - 1B parameters) trained on $\approx$ 20B tokens. We also present a thorough analysis demonstrating that \guide can be combined with knowledge distillation with near additive improvements. Furthermore, we show that applying \guide alone leads to substantially better model quality than applying knowledge distillation by itself. Most importantly, \guide introduces no training or inference overhead and hence any model quality gains from our method are virtually free.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。