揭示MoE模型多语言能力的内部机制，提出定向路由优化方法。
Understanding Multilingualism in Mixture-of-Experts LLMs: Routing Mechanism, Expert Specialization, and Layerwise Steering
- 分析不同语言在专家网络中的路由与使用模式，发现语言家族对齐规律。
- 低资源语言依赖专属专家但性能较差，高资源语言共享专家更高效。
- 通过中间层路由引导提升多语言表现，尤其适用于语系相近的语言对。
Mixture-of-Experts（MoE）架构展现出强大的多语言能力，但其性能提升和跨语言差异背后的内在机制仍不明确。本文系统分析了MoE模型中不同语言的路由行为与专家专业化程度随网络深度的变化。结果表明，多语言处理具有高度结构性：路由行为与语言家族一致，专家使用呈现清晰的分层模式；高资源语言依赖共享专家，而低资源语言则更多使用语言专属专家，尽管其性能较弱。层间干预实验显示，早期和晚期MoE层支持语言特异性处理，中间层则作为语言无关的容量枢纽。基于上述发现，我们提出一种路由引导的推理时调控方法，动态引导中间层路由向主导语言相关的共享专家倾斜，显著提升多语言性能，尤其在语系相近的语言对上效果更优。代码已开源。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) architectures have shown strong multilingual capabilities, yet the internal mechanisms underlying performance gains and cross-language differences remain insufficiently understood. In this work, we conduct a systematic analysis of MoE models, examining routing behavior and expert specialization across languages and network depth. Our analysis reveals that multilingual processing in MoE models is highly structured: routing aligns with linguistic families, expert utilization follows a clear layerwise pattern, and high-resource languages rely on shared experts while low-resource languages depend more on language-exclusive experts despite weaker performance. Layerwise interventions further show that early and late MoE layers support language-specific processing, whereas middle layers serve as language-agnostic capacity hubs. Building on these insights, we propose a routing-guided steering method that adaptively guides routing behavior in middle layers toward shared experts associated with dominant languages at inference time, leading to consistent multilingual performance improvements, particularly for linguistically related language pairs. Our code is available at https://github.com/conctsai/Multilingualism-in-Mixture-of-Experts-LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。