用合成偏好数据微调大模型，提升机器翻译质量
Improving LLMs for Machine Translation Using Synthetic Preference Data
- 通过双模型生成翻译并按质量排序，构建偏好数据集
- 微调后模型在COMET指标上提升0.04和0.02分
- 适合想低成本优化翻译模型的研究者与开发者
大型语言模型已展现出强大的机器翻译能力。本文探讨如何仅用少量易获取的数据资源，提升通用指令微调的大模型在机器翻译上的表现。以斯洛文尼亚语为例，我们使用直接偏好优化（DPO）对GaMS-9B-Instruct模型进行微调，训练数据来自一个程序化筛选并增强的公开数据子集。由于DPO需要成对的质量排名样本，我们利用两个LLM（GaMS-9B-Instruct和EuroLLM-9B-Instruct）分别翻译英文维基百科文章，并结合启发式规则与自动评估指标（如COMET）对结果进行排序。评估表明，微调后的模型优于参与数据生成的两个原始模型。相较于基线模型，其在翻译维基百科文章时的COMET得分分别提升了约0.04和0.02，且更一致地避免语言与格式错误。
原文摘要 · Abstract (English)
Large language models have emerged as effective machine translation systems. In this paper, we explore how a general instruction-tuned large language model can be improved for machine translation using relatively few easily produced data resources. Using Slovene as a use case, we improve the GaMS-9B-Instruct model using Direct Preference Optimization (DPO) training on a programmatically curated and enhanced subset of a public dataset. As DPO requires pairs of quality-ranked instances, we generated its training dataset by translating English Wikipedia articles using two LLMs, GaMS-9B-Instruct and EuroLLM-9B-Instruct. We ranked the resulting translations based on heuristics coupled with automatic evaluation metrics such as COMET. The evaluation shows that our fine-tuned model outperforms both models involved in the dataset generation. In comparison to the baseline models, the fine-tuned model achieved a COMET score gain of around 0.04 and 0.02, respectively, on translating Wikipedia articles. It also more consistently avoids language and formatting errors.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。