用企业翻译记忆库微调大模型,数据越多效果越好。
How Much Data is Enough Data? Fine-Tuning Large Language Models for In-House Translation: Performance Evaluation Across Multiple Dataset Sizes
- 用1千到20万条翻译记忆微调Llama 3,测试不同数据量影响
- 数据量越大翻译质量越高,最大提升13点BLEU和25点COMET
- 小数据集(1k~2k)反而比基线差,适合有大量历史译文的企业
解码器仅有的大语言模型在机器翻译中表现优异,因其能从大规模数据中学习并生成高质量译文。然而,在特定组织所需的术语和风格上常显不足。本研究探索使用特定软件行业组织的翻译记忆库(TMs)微调大语言模型(如Llama 3 8B Instruct)的有效性。实验涵盖五种语言方向(英→巴葡、捷克语、德语、芬兰语、韩语),覆盖1k至207k条翻译段落的数据集规模。针对每组数据分别微调模型,并基于自动评估指标(BLEU、chrF++、TER、COMET)评估性能。结果表明:所有指标随数据量增加而提升;相比基线模型,最大数据集下平均提升13点BLEU和25点COMET。仅用1k或2k样本微调时,性能反而下降;但随着数据量增大,显著改善。研究证明,结合翻译记忆库与大模型可构建定制化翻译系统,提升企业翻译质量并缩短交付周期,为特定领域应用提供有效路径。
原文摘要 · Abstract (English)
Decoder-only LLMs have shown impressive performance in MT due to their ability to learn from extensive datasets and generate high-quality translations. However, LLMs often struggle with the nuances and style required for organisation-specific translation. In this study, we explore the effectiveness of fine-tuning Large Language Models (LLMs), particularly Llama 3 8B Instruct, leveraging translation memories (TMs), as a valuable resource to enhance accuracy and efficiency. We investigate the impact of fine-tuning the Llama 3 model using TMs from a specific organisation in the software sector. Our experiments cover five translation directions across languages of varying resource levels (English to Brazilian Portuguese, Czech, German, Finnish, and Korean). We analyse diverse sizes of training datasets (1k to 207k segments) to evaluate their influence on translation quality. We fine-tune separate models for each training set and evaluate their performance based on automatic metrics, BLEU, chrF++, TER, and COMET. Our findings reveal improvement in translation performance with larger datasets across all metrics. On average, BLEU and COMET scores increase by 13 and 25 points, respectively, on the largest training set against the baseline model. Notably, there is a performance deterioration in comparison with the baseline model when fine-tuning on only 1k and 2k examples; however, we observe a substantial improvement as the training dataset size increases. The study highlights the potential of integrating TMs with LLMs to create bespoke translation models tailored to the specific needs of businesses, thus enhancing translation quality and reducing turn-around times. This approach offers a valuable insight for organisations seeking to leverage TMs and LLMs for optimal translation outcomes, especially in narrower domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。