让多语言专家模型更好理解非英语任务，通过对齐路由模式提升性能。
Routing-Aligned Fine-Tuning for Multilingual Downstream Tasks in Mixture-of-Experts Models
- 基于中间层路由一致性，将任务样本分类并定位关键专家。
- 在标准微调中加入路由对齐损失，显著提升六种语言的下游表现。
- 适合需要多语言适配的大模型应用，尤其关注路由结构的改进。
混合专家（MoE）模型已成为高效扩展大语言模型的主流范式，但将其适配到非英语下游任务仍具挑战性。现有微调方法将MoE视为整体学习器，忽略了预训练过程中形成的异构路由结构。我们在多个MoE模型和下游任务上验证发现，中间层形成语言通用对齐区，路由差异能强预测各语言任务性能差距。基于此，我们提出RA-MoE（路由对齐微调），一个三阶段框架：根据英语与目标语言的正确性将并行任务样本分为四类（cc/ci/ic/ii），识别中间层中的任务相关专家，并在标准SFT基础上引入路由对齐损失，促使ci类样本在目标语言上的路由行为遵循英语任务专家激活模式。在三个MoE模型、三个任务及六种目标语言上的实验表明，RA-MoE持续优于标准SFT及强基线（包括Routing Steering和RISE），且ci比例可作为对齐收益的可靠预测指标。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) models have emerged as a dominant paradigm for efficient LLM scaling, yet adapting them to non-English downstream tasks remains challenging. Existing fine-tuning approaches treat MoE models as monolithic learners, ignoring the heterogeneous routing structure that develops during pretraining. We validate across multiple MoE models and downstream tasks that middle layers form a language-universal alignment zone where routing divergence strongly predicts per-language task performance gaps. Building on this observation, we propose RA-MoE (Routing-Aligned MoE Fine-Tuning), a three-stage framework that categorizes parallel task examples into a four-way taxonomy (cc/ci/ic/ii) based on correctness in English and the target language, identifies task-relevant experts in the middle layers, and augments standard SFT with a routing alignment loss that encourages target-language routing on ci-type examples to follow the English task-expert activation pattern. Experiments across three MoE models, three tasks, and six target languages demonstrate that RA-MoE consistently outperforms standard SFT and strong baselines including Routing Steering and RISE, with the ci proportion of a task-language pair serving as a reliable predictor of alignment benefit.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。