用相似案例动态调整专家路由，提升模型在分布外情况下的适应能力。
Routing by Analogy: kNN-Augmented Expert Assignment for Mixture-of-Experts
- 基于检索过往相似案例，动态优化专家分配策略。
- 在分布外任务上优于零样本基线，媲美耗时微调方法。
- 适合需要鲁棒路由的大型语言模型部署场景。
Mixture-of-Experts（MoE）架构通过参数化路由机制将令牌稀疏地分派给专家子集，实现大语言模型的高效扩展。通常该路由网络训练一次后即被冻结，导致在分布偏移下路由决策脆弱。本文提出kNN-MoE，一种检索增强型路由框架，通过重用记忆库中过去相似案例的最优专家分配来改进路由。该记忆库离线构建，直接优化令牌级路由逻辑以最大化参考集上的似然。关键在于，使用检索到邻居的平均相似度作为置信度驱动的混合系数，当无相关案例时可回退至冻结路由。实验表明，kNN-MoE在分布外任务上优于零样本基线，且与计算开销较大的监督微调方法相当。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) architectures scale large language models efficiently by employing a parametric ``router'' to dispatch tokens to a sparse subset of experts. Typically, this router is trained once and then frozen, rendering routing decisions brittle under distribution shifts. We address this limitation by introducing kNN-MoE, a retrieval-augmented routing framework that reuses locally optimal expert assignments from a memory of similar past cases. This memory is constructed offline by directly optimizing token-wise routing logits to maximize the likelihood on a reference set. Crucially, we use the average similarity of retrieved neighbors as a confidence-driven mixing coefficient, thus allowing the method to fall back to the frozen router when no relevant cases are found. Experiments show that kNN-MoE outperforms the zero-shot baseline and is competitive with computationally intensive supervised fine-tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。