通过渐进式教师模型提升小模型翻译性能，缩小与大模型差距。
Evolving Knowledge Distillation for Lightweight Neural Machine Translation
- 学生模型分阶段从能力递增的教师模型学习
- 在IWSLT-14上学生模型达34.24 BLEU，仅落后大模型0.08
- 适合资源受限设备部署轻量级机器翻译系统
近期神经机器翻译（NMT）进展显著提升了翻译质量，但顶尖模型规模和复杂度增加，给资源受限设备部署带来挑战。知识蒸馏（KD）是压缩模型的有效方法，但在师生模型能力差距较大时效果下降。为此，我们提出进化知识蒸馏（EKD），一种渐进训练框架，学生模型依次从能力逐步增强的教师模型学习。在IWSLT-14、WMT-17和WMT-23基准上的实验表明，EKD在各阶段均实现持续提升。在IWSLT-14上，最终学生模型达到34.24 BLEU，与最强教师模型（34.32 BLEU）的差距仅0.08。其他数据集也呈现相似趋势。结果证明，EKD能有效弥合能力差距，使紧凑模型性能接近大型教师模型。代码与模型见https://github.com/agi-content-generation/EKD。
原文摘要 · Abstract (English)
Recent advancements in Neural Machine Translation (NMT) have significantly improved translation quality. However, the increasing size and complexity of state-of-the-art models present significant challenges for deployment on resource-limited devices. Knowledge distillation (KD) is a promising approach for compressing models, but its effectiveness diminishes when there is a large capacity gap between teacher and student models. To address this issue, we propose Evolving Knowledge Distillation (EKD), a progressive training framework in which the student model learns from a sequence of teachers with gradually increasing capacities. Experiments on IWSLT-14, WMT-17, and WMT-23 benchmarks show that EKD leads to consistent improvements at each stage. On IWSLT-14, the final student achieves a BLEU score of 34.24, narrowing the gap to the strongest teacher (34.32 BLEU) to just 0.08 BLEU. Similar trends are observed on other datasets. These results demonstrate that EKD effectively bridges the capacity gap, enabling compact models to achieve performance close to that of much larger teacher models.Code and models are available at https://github.com/agi-content-generation/EKD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。