用流形幂迭代重设计专家路由，提升模型表达能力
Redesign Mixture-of-Experts Routers with Manifold Power Iteration
- 基于主奇异方向对齐，重构路由矩阵行向量
- 在1B到11B参数规模下验证路由对齐提升模型性能
- 适合研究MoE架构优化与大规模语言模型的开发者
路由器是混合专家（MoE）模型的核心组件。作为专家的代理，路由器矩阵的每一行通过计算与输入的相似性，决定激活哪一组专家。理想情况下，每个路由器行应将专家矩阵编码为代表性向量，使其与输入的点积更准确反映输入与专家的亲和度。然而，目前缺乏设计原则来实现这种信息压缩。本文提出一种基于流形幂迭代（MPI）的路由器重设计方法，将每个路由器行对齐于对应专家的主奇异方向，因为该方向能最有效地描述矩阵特征。具体地，采用“先幂迭代、后投影”的范式：对路由器权重进行幂迭代，再通过投影施加范数约束，确保效率与稳定性。理论证明，MPI能使路由器行收敛至专家的主奇异方向。实验上，在10亿至110亿参数的多种规模模型上预训练，验证了该对齐策略可显著提升MoE模型的有效性。
原文摘要 · Abstract (English)
Router is the cornerstone component to the Mixture-of-Experts models. Serving as expert proxies, the rows of the router matrix compute their similarity to the MoE inputs to determine which subset of experts is activated. Ideally, each router row is designed to encode the expert matrix into this representative vector, such that its dot-product with token can better reflect token-expert affinity. However, there exists no design principles to enforce this condensation. In this paper, we propose to align each router row with the principal singular direction of the associated expert, as this direction provides the most expressive mathematical description of a matrix. Based on this principle, we propose a router redesign with Manifold Power Iteration (MPI). Specifically, it introduces a "Power-then-Retract" paradigm, where a power iteration step is performed on the router weights, followed by a retraction to impose a norm constraint to ensure both efficiency and stability. Theoretically, we show that MPI drives router rows to converge toward the principal singular directions of associated experts. Empirically, we pretrain MoE model across scales from 1B to 11B parameters to confirm that this alignment facilitates more effective MoE models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。