低资源语言在专家模型中路由失衡，持续预训练可有效改善
Mixture of Experts for Low-Resource LLMs
- 用希伯来语测试两种专家模型的路由行为
- 最终层专家使用熵大幅下降，仅少数专家被频繁调用
- 持续预训练能提升路由多样性，适合多语言研究者
Mixture-of-Experts（MoE）架构虽能高效扩展模型，但对低资源语言的专家路由行为仍不明确。本文以希伯来语为形态复杂、资源稀缺的测试语言，分析了两种不同结构的MoE模型——纯Transformer（Qwen3-30B-A3B）和混合Mamba-Transformer（Nemotron-3-Nano-30B-A3B）的路由动态。发现两个预训练模型均出现深层路由坍缩：最终层使用熵急剧下降，大量令牌集中于少数专家，而英语无此现象。在平衡双语数据上进行持续预训练（CPT）显著缓解该不平衡，提升熵值并引导路由向共享、语言无关的专家转移；仅监督微调（SFT）效果较弱。将分析扩展至日语后，亦观察到相似的坍缩特征，表明该现象是预训练中语言代表性不足的系统性结果，而非语言固有属性。路由改进与下游基准性能提升一致，提示路由熵和专家专业化可作为多语言能力的可靠诊断指标。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) architectures enable efficient model scaling, yet expert routing behavior across underrepresented languages remains poorly understood. We analyze routing dynamics in two architecturally distinct MoE models -- a pure Transformer (Qwen3-30B-A3B) and a hybrid Mamba-Transformer (Nemotron-3-Nano-30B-A3B) -- using Hebrew as a morphologically rich, low-resource testbed. Both pre-trained models exhibit \emph{deep-layer routing collapse}: usage entropy drops sharply in final layers and tokens concentrate on a narrow expert subset, a pattern largely absent for English. Continual pre-training (CPT) on balanced bilingual data substantially corrects this imbalance, increasing entropy and shifting routing toward shared, language-agnostic experts; supervised fine-tuning (SFT) alone achieves less complete correction. Extending the analysis to Japanese reveals quantitatively consistent collapse signatures, providing cross-linguistic evidence that the phenomenon is a systematic consequence of pre-training underrepresentation rather than any language-intrinsic property. Routing improvements correlate with consistent downstream benchmark gains, positioning routing entropy and expert specialization as principled diagnostics for multilingual capacity in MoE systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。