arXiv:2411.16300cs.CLcs.AI2024-11被引 8

用320万条跨语言指令,让大模型高效迁移高资源语言能力到低资源语言。

BayLing 2: A Multilingual Large Language Model with Efficient Language Alignment

  • 构建320万条多语言指令数据,实现高-低资源语言间的能力对齐。
  • 在100+语言翻译任务中超越同规模开源模型,20多个低资源语言显著提升。
  • 既保持英语性能,又大幅提升低资源语言表现,适合多语言应用开发。

大型语言模型虽具备强大的生成能力和广泛知识,但主要集中在高资源语言,低资源语言能力较弱。为服务全球100多个语种社区,本文提出BayLing 2,通过语言对齐高效将高资源语言(中文、英文)的生成能力与知识迁移到低资源语言。我们构建了包含320万条指令的数据集,涵盖高资源语言指令和100+语言的跨语言指令,并基于Llama基础模型进行指令微调,得到BayLing-2-7B、BayLing-2-13B和BayLing-2-8B。在100+语言的多语言翻译任务中,BayLing表现优于同规模开源模型;在多语言知识理解基准测试中,超过20个低资源语言取得显著进步,证明其有效知识迁移能力。同时,在英语基准上仍保持高水平性能,兼顾高/低资源语言表现。项目演示、主页、代码与模型均已开放。

原文摘要 · Abstract (English)

Large language models (LLMs), with their powerful generative capabilities and vast knowledge, empower various tasks in everyday life. However, these abilities are primarily concentrated in high-resource languages, leaving low-resource languages with weaker generative capabilities and relatively limited knowledge. Enhancing the multilingual capabilities of LLMs is therefore crucial for serving over 100 linguistic communities worldwide. An intuitive approach to enhance the multilingual capabilities would be to construct instruction data for various languages, but constructing instruction data for over 100 languages is prohibitively costly. In this paper, we introduce BayLing 2, which efficiently transfers generative capabilities and knowledge from high-resource languages to low-resource languages through language alignment. To achieve this, we constructed a dataset of 3.2 million instructions, comprising high-resource language instructions (Chinese and English) and cross-lingual instructions for 100+ languages and performed instruction tuning based on the dataset to facilitate the capability transfer between languages. Using Llama as the foundation model, we developed BayLing-2-7B, BayLing-2-13B, and BayLing-2-8B, and conducted a comprehensive evaluation of BayLing. For multilingual translation across 100+ languages, BayLing shows superior performance compared to open-source models of similar scale. For multilingual knowledge and understanding benchmarks, BayLing achieves significant improvements across over 20 low-resource languages, demonstrating its capability of effective knowledge transfer from high-resource to low-resource languages. Furthermore, results on English benchmarks indicate that BayLing maintains high performance in highresource languages while enhancing the performance in low-resource languages. Demo, homepage, code and models of BayLing are available.

大模型多语言知识迁移低资源语言

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。