提出预注意力专家预测，提升MoE模型推理速度与精度。
Pre-Attention Expert Prediction and Prefetching for Mixture-of-Experts Large Language Models
- 用前注意力层激活值，通过两个线性函数实现精准专家预测。
- 在三个模型上准确率达93%~97.6%，较现有方法提升约15%。
- 轻量设计支持首层预取，适合部署于资源受限的MoE推理场景。
混合专家（MoE）大语言模型在保持较低推理成本的同时实现了高效扩展。由于仅激活部分专家，已有工作提出专家预测与缓存机制以提前加载专家，加速推理。然而，现有方法依赖前一层激活进行预测，准确率低且未优化首层。若引入复杂结构或独立训练网络以提升预测性能，则带来高计算开销。本文提出预注意力专家预测方法，实现高精度且轻量的专家预取。核心洞察是大模型中某些函数具有排序保持特性，即通过简单线性函数匹配专家选择排名是可行的。因此，我们利用同一层注意力块前的激活值，结合两个线性函数与排序感知损失，实现精确预测，并支持首层预取。所提出的轻量级预注意力路由在DeepSeek V2 Lite上达到93.03%准确率，Qwen3-30B上达94.69%，Phi-mini-MoE上达97.62%，相比当前最优方法绝对准确率提升约15%。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) Large Language Models (LLMs) efficiently scale-up the model while keeping relatively low inference cost. As MoE models only activate part of the experts, related work has proposed expert prediction and caching methods to prefetch the experts for faster inference. However, existing approaches utilize the activations from the previous layer for prediction, incurring low accuracy and leave the first layer unoptimized. Applying complex layers or even training standalone networks for better prediction introduces high computation overhead. In this paper, we propose pre-attention expert prediction to achieve accurate and lightweight expert prefetching. The key insight is that some functions in LLMs are ranking-preserving, indicating that matching the ranking of selected experts using simple linear functions is possible. Therefore, we utilize the activations before the attention block in the same layer with 2 linear functions and ranking-aware loss to achieve accurate prediction, which also supports prefetching in the first layer. Our lightweight, pre-attention expert routers achieve 93.03% accuracy on DeepSeek V2 Lite, 94.69% on Qwen3-30B, and 97.62% on Phi-mini-MoE, showing about 15% improvement on absolute accuracy over the state-of-the-art methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。