发现MoE模型中语言专精仅在最后层出现，据此提出高效低参适配新方法。
Leveraging Routing Dynamics in Mixture-of-Experts Models for Efficient Language Adaptation
- 只更新最终层的特定专家与共享专家，实现参数高效适配
- 在MultiBLiMP和Belebele上性能接近全层微调，参数更新少于2%
- 揭示语言间词汇重叠影响路由行为，指导低资源多语适配
Mixture-of-Experts（MoE）模型广泛用于扩展语言模型，但其在多语言场景下的专家路由行为与适应机制仍不清晰。本文研究在多语言语料上持续预训练以英语为中心的MoE模型时的多语言路由动态，分析不同语言下专家使用差异。发现持续多语言预训练导致早期和中间层路由分散、语言无关，语言专精主要在最后一层出现。同时，语言间词级词汇重叠对路由有重要影响。基于此，我们提出一种参数高效的适配策略，仅更新最终层的语言特定与共享专家。在MultiBLiMP和Belebele数据集上的实验表明，该方法在性能与效率间取得良好平衡，性能媲美全层微调，参数更新量不足2%。研究结果揭示了多语言持续预训练中语言专精的演化路径，并为低资源多语言适配提供实用指导。代码已开源。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) models are widely used to scale language models, yet their expert routing behavior and adaptation in a multilingual setting remain underexplored. In this work, we study multilingual routing dynamics during continual pre-training of an English-centric MoE model on a multilingual corpus, analyzing how expert usage varies across languages. We find that continual multilingual pre-training leads to diffused, language-agnostic routing in early and middle layers, with language specialization primarily emerging in the final layers. We also show that token-level vocabulary overlap between languages plays an important role in how languages are routed. Motivated by these findings, we propose a parameter-efficient adaptation strategy that updates language-specific and shared experts in the final MoE layers. Experiments on MultiBLiMP and Belebele show that our method achieves a strong performance-efficiency trade-off, attaining competitive performance relative to fine-tuning complete final layers, while updating less than 2% of the parameters. Overall, our findings provide insights into where and how language specialization emerges in MoEs during continual pre-training and provide practical insights for low-resource multilingual adaptation. Our code is available at https://github.com/aditi184/moe-routing-adaptation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。