用260亿词元数据微调，让开源大模型更好支持20种非洲语言
AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languages
- 在260亿词元上持续预训练，混合数学、代码和合成翻译数据提升性能
- 加入数学与代码数据后，模型推理能力显著提升，尤其在多语言评测中表现更好
- 模型架构比规模更重要，任务对齐数据+稳健架构才是关键
大型语言模型（LLMs）日益多语化，但开源模型在性能上仍落后于专有系统，尤其在非洲语言上差距明显。持续预训练（CPT）为语言适配提供了可行路径，但在数学推理等高难度任务上的提升仍有限，这主要源于低资源语言语料库的领域覆盖不均和任务相关知识缺失。我们提出 exttt{AfriqueLLM}，一套通过在260亿词元上进行持续预训练，适配20种非洲语言的开源大模型。我们在五个不同规模和架构的基座模型（包括 Llama 3.1、Gemma 3、Qwen 3）上展开全面实证研究，系统分析数据组成对下游性能的影响。特别地，我们测试了包含数学、代码和合成翻译数据的不同混合方案，并在多种多语言基准上评估结果。结果显示，数据组成是持续预训练效果的主要驱动因素。加入数学、代码和合成翻译数据能带来稳定提升，包括在推理类评测中。在固定架构下，更大模型通常表现更好，但跨模型族比较时，架构选择比规模更具决定性。此外，基座模型的多语言表现不能可靠预测持续预训练后的结果；稳健架构结合任务对齐数据才是更可靠的方案。最终，最佳模型还提升了长上下文处理能力，包括文档级翻译。模型与代码已发布于 [Huggingface](https://huggingface.co/collections/McGill-NLP/afriquellm) 和 [Github](https://github.com/McGill-NLP/AfriqueLLM)。
原文摘要 · Abstract (English)
Large language models (LLMs) are increasingly multilingual, yet open models continue to underperform relative to proprietary systems, with the gap most pronounced for African languages. Continued pre-training (CPT) offers a practical route to language adaptation, but improvements on demanding capabilities such as mathematical reasoning often remain limited. This limitation is driven in part by the uneven domain coverage and missing task-relevant knowledge that characterize many low-resource language corpora. We present \texttt{AfriqueLLM}, a suite of open LLMs adapted to 20 African languages through CPT on 26B tokens. We perform a comprehensive empirical study across five base models spanning sizes and architectures, including Llama 3.1, Gemma 3, and Qwen 3, and systematically analyze how CPT data composition shapes downstream performance. In particular, we vary mixtures that include math, code, and synthetic translated data, and evaluate the resulting models on a range of multilingual benchmarks. Our results identify data composition as the primary driver of CPT gains. Adding math, code, and synthetic translated data yields consistent improvements, including on reasoning-oriented evaluations. Within a fixed architecture, larger models typically improve performance, but architectural choices dominate scale when comparing across model families. Moreover, strong multilingual performance in the base model does not reliably predict post-CPT outcomes; robust architectures coupled with task-aligned data provide a more dependable recipe. Finally, our best models improve long-context performance, including document-level translation. Models and code have been released on [Huggingface](https://huggingface.co/collections/McGill-NLP/afriquellm) and [Github](https://github.com/McGill-NLP/AfriqueLLM).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。