arXiv:2605.04952cs.LG2026-05

提出新型路由机制,让细粒度专家模型更快更准。

Adaptive Inverted-Index Routing for Granular Mixtures-of-Experts

论文配图:Adaptive Inverted-Index Routing for Granular Mixtures-of-Experts
图 1 · 摘自论文原文
  • 两阶段路由:先用向量量化粗筛专家,再精准评分
  • 在细粒度专家设置下,性能优于现有方法
  • 无需修改模型结构,可直接替换原有路由模块

Mixture-of-experts (MoE) 模型通过仅激活每令牌的子集专家实现可扩展的 Transformer 架构。最新研究表明,专家越细粒度(即更多小专家而非少数大专家),性能越好。然而,这种设置显著增加路由开销,可能主导计算成本。我们提出自适应倒排索引路由(AIR-MoE),一种基于向量量化(VQ)的倒排索引启发式路由架构。第一阶段,通过将令牌分配给 VQ 编码字来粗略筛选候选专家集;第二阶段,仅在该候选集上计算精确路由分数。该两阶段过程近似真实 top-k 路由,避免全专家评分,且与先前工作不同,不对专家参数施加结构约束。AIR-MoE 可作为标准路由器的即插即用替代品,无需修改模型架构或损失函数。我们进一步提供了 AIR-MoE 实现的最小质量召回率下界,揭示其内部机制。实验表明,在细粒度 MoE 设置下,AIR-MoE 的性能优于现有路由方法。

原文摘要 · Abstract (English)

Mixture-of-experts (MoE) models enable scalable transformer architectures by activating only a subset of experts per token. Recent evidence suggests that performance improves with increasingly granular experts, i.e., many small experts instead of a few large ones. However, this regime substantially increases routing cost, which can dominate computation. We introduce adaptive inverted-index routing for MoE (AIR-MoE), an inverted-index-inspired routing architecture based on vector quantization (VQ). In a first stage, AIR-MoE performs coarse shortlisting by assigning tokens to VQ codewords to construct a candidate set of experts. In a second stage, fine scoring computes exact routing scores restricted to this shortlist. This two-stage procedure approximates true top-k routing while avoiding full expert scoring and, in contrast to prior work, imposing no structural constraints on expert parameters. AIR-MoE serves as a drop-in replacement for standard routers and requires no modifications to the model architecture or loss function. We further provide a lower bound on the mass recall achieved by AIR-MoE that yields insights into its inner workings. Empirically, we demonstrate that AIR-MoE achieves improved performance compared to existing routing approaches in granular MoE settings.

MoE路由优化向量量化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。