动态调整知识蒸馏难度，提升大模型翻译压缩效果
Self-Evolution Knowledge Distillation for LLM-based Machine Translation
- 按词元难易度动态融合教师分布与真实标签
- 在WMT22上平均提升1.4个SacreBLEU点
- 适合追求高效小模型翻译的开发者
知识蒸馏（KD）在将大模型知识迁移至小模型方面展现出巨大潜力。然而，现有大语言模型的蒸馏策略通常对每个词元无差别地最小化学生与教师模型的输出分布差异，忽略了词元间存在的不平衡性及其转移难度差异。为此，我们提出一种名为自进化知识蒸馏（Self-Evolution KD）的新策略。该方法的核心是将教师模型分布与真实标签的独热分布动态结合为学生模型的先验知识，以促进蒸馏过程，并根据词元学习难度调整先验知识比例，充分挖掘教师模型潜力。实验结果表明，该方法在WMT22测试集四个翻译方向上平均提升约1.4个SacreBLEU点。进一步分析显示，性能提升源于教师模型更有效的知识传递，验证了我们的假设。
原文摘要 · Abstract (English)
Knowledge distillation (KD) has shown great promise in transferring knowledge from larger teacher models to smaller student models. However, existing KD strategies for large language models often minimize output distributions between student and teacher models indiscriminately for each token. This overlooks the imbalanced nature of tokens and their varying transfer difficulties. In response, we propose a distillation strategy called Self-Evolution KD. The core of this approach involves dynamically integrating teacher distribution and one-hot distribution of ground truth into the student distribution as prior knowledge, which promotes the distillation process. It adjusts the ratio of prior knowledge based on token learning difficulty, fully leveraging the teacher model's potential. Experimental results show our method brings an average improvement of approximately 1.4 SacreBLEU points across four translation directions in the WMT22 test sets. Further analysis indicates that the improvement comes from better knowledge transfer from teachers, confirming our hypothesis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。