通过动态分组语言并分配专家模块，提升多语言大模型性能。
Group then Scale: Dynamic Mixture-of-Experts Multilingual Language Model
- 按语言相似性分组，为相似语言分配专用专家模块。
- 在18至128种语言上实验，用更少参数显著提升多语言表现。
- 支持新语言快速适配，减少对旧知识的依赖和干扰。
多语言大模型面临'多语言诅咒'问题：多种语言间竞争导致性能下降，主要源于容量有限及不相似语言间的负迁移。为此，我们提出一种动态分组并扩展参数的方法，在提升相似语言正迁移的同时缓解语言间竞争。具体而言，先在单语语料上微调模型，识别各层参数偏差，并量化语言相似性；参数偏差大的层被扩展为混合专家（MoE）结构，每个专家对应一组相似语言。在18到128种语言上的实验表明，该方法有效降低语言间负迁移，以较少参数显著提升多语言性能。语言分组专家机制还提升了新语言适应能力，减少了对已有多语言知识的推理负担。
原文摘要 · Abstract (English)
The curse of multilinguality phenomenon is a fundamental problem of multilingual Large Language Models (LLMs), where the competition between massive languages results in inferior performance. It mainly comes from limited capacity and negative transfer between dissimilar languages. To address this issue, we propose a method to dynamically group and scale up the parameters of multilingual LLM while boosting positive transfer among similar languages. Specifically, the model is first tuned on monolingual corpus to determine the parameter deviation in each layer and quantify the similarity between languages. Layers with more deviations are extended to mixture-of-experts layers to reduce competition between languages, where one expert module serves one group of similar languages. Experimental results on 18 to 128 languages show that our method reduces the negative transfer between languages and significantly boosts multilingual performance with fewer parameters. Such language group specialization on experts benefits the new language adaptation and reduces the inference on the previous multilingual knowledge learned.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。