用多候选翻译提升低资源语言翻译效果，改善模型偏差
Multi-Hypothesis Distillation of Multilingual Neural Translation Models for Low-Resource Languages
- 通过生成多个翻译候选，更全面地传递教师模型分布
- 在低资源语言上，采样方法提升语料多样性，翻译质量略降但整体性能更好
- 适合关注低资源语言、模型偏差与数据多样性的研究者
本文研究多语言预训练编码器-解码器翻译模型的序列级知识蒸馏。我们指出，教师模型输出分布蕴含的信息远超束搜索所得的单一最优译文，提出多假设蒸馏（MHD）方法，为每条源句生成多个目标翻译，以更完整呈现教师分布并扩展学生模型接触的目标前缀范围。利用束搜索的n-best列表指导学生学习，并探索其他解码方式以解决低变异性及罕见词覆盖不足的问题。实验表明，对低资源语言而言，虽采样方法相比束搜索略降低翻译质量，但显著提升生成语料的多样性与词汇丰富性，最终增强学生模型性能，并缓解知识蒸馏中常见的性别偏见放大问题。
原文摘要 · Abstract (English)
This paper explores sequence-level knowledge distillation (KD) of multilingual pre-trained encoder-decoder translation models. We argue that the teacher model's output distribution holds valuable insights for the student, beyond the approximated mode obtained through beam search (the standard decoding method), and present Multi-Hypothesis Distillation (MHD), a sequence-level KD method that generates multiple translations for each source sentence. This provides a larger representation of the teacher model distribution and exposes the student model to a wider range of target-side prefixes. We leverage $n$-best lists from beam search to guide the student's learning and examine alternative decoding methods to address issues like low variability and the under-representation of infrequent tokens. For low-resource languages, our research shows that while sampling methods may slightly compromise translation quality compared to beam search based approaches, they enhance the generated corpora with greater variability and lexical richness. This ultimately improves student model performance and mitigates the gender bias amplification often associated with KD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。