arXiv:2510.14853cs.CL2025-10被引 3

不依赖外部数据,在线动态优化MoE模型专家路由,提升推理性能。

Rewiring Experts on the Fly:Continuous Rerouting for Better Online Adaptation in Mixture-of-Expert models

  • 基于生成内容自监督,实时调整MoE专家选择策略。
  • 在HumanEval上提升5.5%,集成自一致性时平均增益6%。
  • 轻量级插件式设计,兼容现有测试时扩展技术。

Mixture-of-Experts (MoE) 模型通过稀疏专家激活实现高效扩展,但在部署中常因分布偏移导致路由决策不佳。现有测试时适应方法多针对密集模型,需外部数据,难以应用于MoE架构。本文提出一种无需外部数据的在线测试时框架,仅基于输入上下文实时优化MoE路由。方法分两阶段:预填充阶段及后续周期性更新中,利用已生成序列进行自监督优化路由;随后正常生成文本并保留修改后的路由器直至下一次调整。通过轻量级加性向量仅更新选定层的路由器logits,保持计算效率且避免过拟合。实验表明,在挑战性推理任务上持续提升性能,如OLMoE在HumanEval上提升5.5%;其插件特性可自然融合现有测试时缩放技术,与自一致性结合时在DeepSeek-V2-Lite上实现6%平均增益。

原文摘要 · Abstract (English)

Mixture-of-Experts (MoE) models achieve efficient scaling through sparse expert activation, but often suffer from suboptimal routing decisions due to distribution shifts in deployment. While existing test-time adaptation methods could potentially address these issues, they primarily focus on dense models and require access to external data, limiting their practical applicability to MoE architectures. However, we find that, instead of relying on reference data, we can optimize MoE expert selection on-the-fly based only on input context. As such, we propose \textit{a data-free, online test-time framework} that continuously adapts MoE routing decisions during text generation without external supervision or data. Our method cycles between two phases: During the prefill stage, and later in regular intervals, we optimize the routing decisions of the model using self-supervision based on the already generated sequence. Then, we generate text as normal, maintaining the modified router until the next adaption. We implement this through lightweight additive vectors that only update router logits in selected layers, maintaining computational efficiency while preventing over-adaptation. The experimental results show consistent performance gains on challenging reasoning tasks while maintaining robustness to context shifts. For example, our method achieves a 5.5\% improvement on HumanEval with OLMoE. Furthermore, owing to its plug-and-play property, our method naturally complements existing test-time scaling techniques, e.g., achieving 6\% average gains when incorporated with self-consistency on DeepSeek-V2-Lite.

MoE在线适应自监督推理优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。