提出L2R路由框架，提升专家模型的路由稳定性和专家区分度。
L2R: Low-Rank and Lipschitz-Controlled Routing for Mixture-of-Experts
- 在低秩潜空间中进行路由，结合饱和内积评分控制函数平滑性
- 在语言和视觉MoE任务中均显著改善路由几何与模型性能
- 适合追求路由稳定性与专家表达力的大型模型研究者
Mixture-of-Experts（MoE）模型通过条件激活少量专家来扩展神经网络，其中路由器在决定专家专业化和整体性能中起核心作用。然而，许多现代MoE系统仍采用原始高维表示空间中的线性路由器，导致表示不匹配、角度集中和尺度敏感评分等问题，共同削弱了路由可区分性与专家稳定专业化。本文提出低秩与Lipschitz可控路由（L2R），一种统一的路由框架，同时重塑路由空间与评分几何。L2R在共享低秩潜路由空间中执行专家分配，并引入饱和内积评分（SIPS）以显式控制路由函数的Lipschitz性质，实现更平滑稳定的路由几何。此外，L2R引入参数高效的多锚点路由机制以增强专家表达能力。在基于OLMoE的语言MoE模型和ImageNet上的视觉MoE设置中进行的大量实验表明，L2R持续改进路由几何、专家区分度与整体模型性能。代码将公开。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) models scale neural networks by conditionally activating a small subset of experts, where the router plays a central role in determining expert specialization and overall model performance. However, many modern MoE systems still adopt linear routers in raw high-dimensional representation spaces, where representation mismatch, angular concentration, and scale-sensitive scoring can jointly undermine routing discriminability and stable expert specialization. In this work, we propose Low-rank & Lipschitz-controlled Routing (L2R), a unified routing framework that reshapes both the routing space and scoring geometry. L2R performs expert assignment in a shared low-rank latent routing space and introduces Saturated Inner-Product Scoring (SIPS) to explicitly control the Lipschitz behavior of routing functions, yielding smoother and more stable routing geometry. In addition, L2R incorporates a parameter-efficient multi-anchor routing mechanism to enhance expert expressiveness. Extensive experiments on an OLMoE-based language MoE model and a vision MoE setting on ImageNet demonstrate that L2R consistently improves routing geometry, expert discrimination, and overall model performance. Code will be released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。