通过动态重路由专家激活，提升大模型生成速度而不降精度。
Opportunistic Expert Activation: Batch-Aware Expert Routing for Faster Decode Without Retraining
- 基于批处理感知的专家路由，让token共享已加载专家。
- 在16批大小下，推理延迟降低39%至15%。
- 适合追求高效推理的部署场景，无需重新训练。
越来越多的大语言模型采用混合专家（MoE）架构，用一组专家替代前馈层，每个标记仅激活其中一小部分专家。在自回归生成过程中，即使批量大小适中，这些模型也常进入内存受限状态，因为平均专家负载的增长速度慢于等效的密集前馈层。因此，MoE的延迟主要由激活的专家数量决定。我们提出一种动态重路由机制，以减少激活专家数（从而降低解码延迟），同时保持相近的性能。最优方案采用批处理感知路由，使标记能复用同一批次中其他标记所依赖的、已载入内存的专家。我们在Qwen3-30B和Qwen3-235B模型上评估该方法，批量大小为16。在无统计显著精度损失的情况下，摩尔层解码延迟分别降低39%和15%。
原文摘要 · Abstract (English)
An increasing number of LLMs employ Mixture-of-Experts (MoE) architectures where the feed-forward layer is replaced by a pool of experts and each token only activates a small subset of them. During autoregressive generation, these models often enter a memory-bound regime even for moderate batch sizes because the average expert load grows more slowly than in an equivalent dense feedforward layer. Consequently, MoE latency is governed by the number of activated experts. We introduce a framework for dynamically re-routing token-to-expert mapping to lower this number (and thus, the decode latency) while preserving a comparable quality. Our best results use a batch-aware routing that works by having tokens piggyback experts that have already been loaded into memory due to being crucial to other tokens within the same batch. Empirically, we evaluate our method on the Qwen3-30B and Qwen3-235B models with a batch size of $16$. Without any statistically significant loss in accuracy, our approach achieves latency reductions of $39\%$ and $15\%$ in the MoE layer decode latency, respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。