用英语教育文本+非洲语料训练大模型，提升非洲语言理解能力
Lugha-Llama: Adapting Large Language Models for African Languages
- 混合非洲语料与高质量英文教育文本进行训练
- 在IrokoBench上知识问答任务表现优于同类模型
- 适合关注低资源语言与AI公平性的研究者
大语言模型在多种自然语言任务中表现优异，但对低资源语言（尤其是非洲语言）识别能力较弱，因这些语言在大规模训练语料中代表性不足。本文探索如何将大模型适配至低资源非洲语言。实验发现，将精选的非洲语言数据与高质量英文教育文本结合，可显著提升模型在非洲语言上的性能。在挑战性IrokoBench基准上，我们的模型在知识密集型多选题（AfriMMLU）任务中持续领先同类规模基线。在跨语言问答基准AfriQA上，模型性能较基线提升超过10%。通过将200M tokens英文数据翻译为斯瓦希里语并分析，发现英文内容是性能提升的关键因素。论文已开源模型与数据，以推动非洲语言研究。
原文摘要 · Abstract (English)
Large language models (LLMs) have achieved impressive results in a wide range of natural language applications. However, they often struggle to recognize low-resource languages, in particular African languages, which are not well represented in large training corpora. In this paper, we consider how to adapt LLMs to low-resource African languages. We find that combining curated data from African languages with high-quality English educational texts results in a training mix that substantially improves the model's performance on these languages. On the challenging IrokoBench dataset, our models consistently achieve the best performance amongst similarly sized baselines, particularly on knowledge-intensive multiple-choice questions (AfriMMLU). Additionally, on the cross-lingual question answering benchmark AfriQA, our models outperform the base model by over 10%. To better understand the role of English data during training, we translate a subset of 200M tokens into Swahili language and perform an analysis which reveals that the content of these data is primarily responsible for the strong performance. We release our models and data to encourage future research on African languages.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。