让小模型更准确模仿大模型，通过学习其训练时的分布变化。
Delta Knowledge Distillation for Large Language Models
- 显式建模教师模型微调带来的分布变化，指导学生学习。
- 在ROUGE指标上显著提升学生模型性能，保留更多教师知识。
- 适合需要高精度压缩大语言模型的场景。
知识蒸馏（KD）是一种广泛采用的压缩大型神经网络的方法，通过将大型教师模型的知识迁移到小型学生模型中实现。在大语言模型背景下，基于标记级别的知识蒸馏通常通过最小化学生输出分布与教师输出分布之间的KL散度来实现，表现出良好的实证性能。然而，先前工作假设学生与教师的输出分布共享相同最优表示空间，这一前提在许多情况下可能不成立。为解决此问题，我们提出了一种新的标记级别蒸馏方法——Delta知识蒸馏（Delta-KD），该方法通过显式保留教师模型在监督微调（SFT）过程中引入的分布变化Δ，鼓励学生模型逼近这一最优表示空间。实验结果表明，在ROUGE指标上，Delta-KD显著提升了学生模型的表现，同时保留了更多教师模型的知识。
原文摘要 · Abstract (English)
Knowledge distillation (KD) is a widely adopted approach for compressing large neural networks by transferring knowledge from a large teacher model to a smaller student model. In the context of large language models, token level KD, typically minimizing the KL divergence between student output distribution and teacher output distribution, has shown strong empirical performance. However, prior work assumes student output distribution and teacher output distribution share the same optimal representation space, a premise that may not hold in many cases. To solve this problem, we propose Delta Knowledge Distillation (Delta-KD), a novel extension of token level KD that encourages the student to approximate an optimal representation space by explicitly preserving the distributional shift Delta introduced during the teacher's supervised finetuning (SFT). Empirical results on ROUGE metrics demonstrate that Delta KD substantially improves student performance while preserving more of the teacher's knowledge.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。