用大模型循环蒸馏，零样本实现低资源语言高质量翻译
CycleDistill: Bootstrapping Machine Translation using LLMs with Cyclical Distillation
- 通过零/少样本翻译生成合成语料，迭代优化模型
- 仅需1-4个示例，首轮翻译效果提升20-30 chrF点
- 适合无平行语料的低资源语言翻译任务
大型语言模型（LLMs）虽具备少量示例下的机器翻译能力，但通常仍不及在平行语料上训练的专用翻译系统。然而，对于低资源语言，平行语料往往稀缺或缺失。本文提出CycleDistill，一种基于大模型和少样本翻译的自举方法，以构建高质量机器翻译系统。该方法通过零样本或少样本翻译，从单语语料中迭代生成合成平行语料，并用于微调生成该数据的模型。实验聚焦三种印度语言，仅依赖单语语料，无需额外平行语料（除1至4个少样本示例外），在首次迭代中平均提升20-30 chrF点，显著优于基线模型。同时研究了蒸馏过程中使用softmax激活的影响，观察到翻译质量略有提升。
原文摘要 · Abstract (English)
Large language models (LLMs), despite their ability to perform few-shot machine translation (MT), often lag behind dedicated MT systems trained on parallel corpora, which are crucial for high quality machine translation (MT). However, parallel corpora are often scarce or non-existent for low-resource languages. In this paper, we propose CycleDistill, a bootstrapping approach leveraging LLMs and few-shot translation to obtain high-quality MT systems. CycleDistill involves iteratively generating synthetic parallel corpora from monolingual corpora via zero- or few-shot MT, which is then used to fine-tune the model that was used for generating said data for MT. CycleDistill does not need parallel corpora beyond 1 to 4 few-shot examples, and in our experiments focusing on three Indian languages, by relying solely on monolingual corpora, it can achieve high-quality machine translation, improving upon a few-shot baseline model by over 20-30 chrF points on average in the first iteration. We also study the effect of leveraging softmax activations during the distillation process and observe mild improvements in translation quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。