通过多层上下文融合提升专家选择一致性，让模型更稳定高效。
Multi-level context Modeling for consistent expert selection in Mixture-of-Experts
- 融合跨层语义与局部交互，构建更完整的上下文表示
- 在语言建模和理解任务中显著提升路由一致性与性能
- 适合需要稳定专家路由的大型Transformer模型研究者
Mixture-of-Experts（MoE）通过将令牌路由到少量专家实现Transformer模型的高效扩展。然而，现有路由器通常基于浅层或孤立的令牌表示进行专家选择，导致各层间路由决策不稳定且语义不一致。本文从表征视角重新审视专家选择，识别出上下文不完整是限制专家专业化的主要瓶颈。为此，提出多层上下文融合MoE（MCF-MOE），通过整合跨层语义聚合与局部令牌级交互的互补信号，构建上下文感知表示，实现更丰富、更一致的专家选择。在语言建模与理解基准上的实验表明，MCF-MOE持续优于强基线模型，在路由一致性与下游性能上均有提升，凸显了上下文完整性在专家路由中的重要性。代码已公开于https://anonymous.4open.science/r/MCFMOE。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) enables efficient scaling of Transformer models by routing tokens to a small subset of experts. However, existing routers typically condition expert selection on shallow or isolated token representations, which often produce unstable and semantically inconsistent routing decisions across layers. In this work, we revisit expert selection from a representation perspective and identify context incompleteness as a key bottleneck limiting effective expert specialization. To address this issue, we propose Multi-level Context Fusion MOE (MCF-MOE), a framework that constructs context-aware representations by integrating complementary signals from cross-layer semantic aggregation and local token-level interactions, enabling more informative and consistent expert selection. Experiments on language modeling and understanding benchmarks demonstrate that MCF-MOE consistently improves routing consistency and downstream performance over strong MoE baselines, highlighting the importance of contextual completeness in expert routing. The code is available at https://anonymous.4open.science/r/MCFMOE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。