Babel覆盖全球90%人口的25种语言，填补开源多语言模型空白。
Babel: Open Multilingual Large Language Models Serving Over 90% of Global Speakers
- 通过层扩展技术扩大参数量，突破传统持续预训练局限
- Babel-83B-Chat在多语言任务上达到商用模型水平
- 专注高覆盖率、低资源语言，适合多语言应用开发者
大型语言模型（LLMs）已彻底改变自然语言处理（NLP），但开源多语言LLM仍十分稀缺，现有模型通常仅覆盖高资源语言，忽略大量使用但资源匮乏的语言。为解决这一不平衡问题，我们提出 $ exttt{Babel}$，一个开源多语言LLM，覆盖按使用者数量排名前25的语言，支持超过90%的全球人口，并包含许多其他开源多语言模型忽略的语言。不同于传统的持续预训练方法，Babel采用层扩展技术提升参数量，从而提高性能上限。我们推出了两个版本：$ exttt{Babel-9B}$，适用于高效推理与微调；$ exttt{Babel-83B}$，为开源多语言LLM树立新标准。在多语言任务上的广泛评估表明，其性能优于同等规模的开源LLM。此外，利用开源监督微调数据集，Babel-9B-Chat在100亿级模型中表现领先，而Babel-83B-Chat在多语言任务上达到商业模型水平。
原文摘要 · Abstract (English)
Large language models (LLMs) have revolutionized natural language processing (NLP), yet open-source multilingual LLMs remain scarce, with existing models often limited in language coverage. Such models typically prioritize well-resourced languages, while widely spoken but under-resourced languages are often overlooked. To address this disparity, we introduce $\texttt{Babel}$, an open multilingual LLM that covers the top 25 languages by number of speakers, supports over 90% of the global population, and includes many languages neglected by other open multilingual LLMs. Unlike traditional continue pretraining approaches, Babel expands its parameter count through a layer extension technique that elevates Babel's performance ceiling. We introduce two variants: $\texttt{Babel-9B}$, designed for efficient inference and fine-tuning, and $\texttt{Babel-83B}$, which sets a new standard for open multilingual LLMs. Extensive evaluations on multilingual tasks demonstrate its superior performance compared to open LLMs of comparable size. In addition, using open-source supervised fine-tuning datasets, Babel achieves remarkable performance, with Babel-9B-Chat leading among 10B-sized LLMs and Babel-83B-Chat setting a new standard for multilingual tasks, reaching the same level of commercial models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。