用机器翻译提升低资源语言模型性能，仅用少量数据超越现有主流模型。
Multilingual Language Model Pretraining using Machine-translated Data
- 用高质量英文数据经机器翻译构建多语言语料库，扩充非英语训练资源。
- 1.3B参数模型在9个非英语任务上媲美甚至超过闭源大模型，仅用十分之一数据。
- 加入不足5%的翻译数据即可显著提升阿拉伯语等5种语言的理解与推理能力。
英语等高资源语言支持高质量大语言模型的预训练，但多数其他语言仍表现不佳，可能源于多语言预训练语料的质量与多样性不足。本文发现，从单一高质量源语言（英语）进行机器翻译，可显著提升多语言大模型的预训练效果。我们将高质量英文网页数据集FineWeb-Edu翻译成九种语言，生成1.7万亿词的语料库TransWebEdu，并基于此从零训练一个1.3B参数的模型TransWebLLM。在九个非英语推理任务中，TransWebLLM的表现达到或超过使用闭源数据训练的SOTA模型（如Llama3.2、Qwen2.5、Gemma），且所用数据量少一个数量级。实验表明，仅添加不到5%的TransWebEdu作为领域特定预训练数据，即可在阿拉伯语、意大利语、印尼语、斯瓦希里语和威尔士语的理解与常识推理任务上达到新SOTA。为促进可复现性，我们已将语料库、模型及训练流程以开源许可证发布。
原文摘要 · Abstract (English)
High-resource languages such as English, enables the pretraining of high-quality large language models (LLMs). The same can not be said for most other languages as LLMs still underperform for non-English languages, likely due to a gap in the quality and diversity of the available multilingual pretraining corpora. In this work, we find that machine-translated texts from a single high-quality source language can contribute significantly to the pretraining quality of multilingual LLMs. We translate FineWeb-Edu, a high-quality English web dataset, into nine languages, resulting in a 1.7-trillion-token dataset, which we call TransWebEdu and pretrain a 1.3B-parameter model, TransWebLLM, from scratch on this dataset. Across nine non-English reasoning tasks, we show that TransWebLLM matches or outperforms state-of-the-art multilingual models trained using closed data, such as Llama3.2, Qwen2.5, and Gemma, despite using an order of magnitude less data. We demonstrate that adding less than 5% of TransWebEdu as domain-specific pretraining data sets a new state-of-the-art in Arabic, Italian, Indonesian, Swahili, and Welsh understanding and commonsense reasoning tasks. To promote reproducibility, we release our corpus, models, and training pipeline under Open Source Initiative-approved licenses.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。