动态调整专家选择,让混合专家模型推理更快更省带宽。
AcceptMoE: Commitment-Weighted Self-Sizing Verifier Expert Sets for Efficient MoE Speculative Decoding

- 根据承诺概率自动调节每块验证的可用专家数,无需手动设预算。
- 在单批处理下吞吐量达基线1.29倍,显存外移时提速2.06倍。
- 适合追求高吞吐、低通信开销的MoE模型部署场景。
推测解码通过一次目标模型前向传播验证草案令牌树。对于混合专家(MoE)目标模型,平行验证可能激活所有树节点选中的专家并集,尽管只有少数节点最终被采纳。因此,令牌数量、激活专家并集大小和专家权重流量是不同的成本指标:减少令牌负载未必能成比例缩小专家并集,且在卸载场景下,传输流量还依赖缓存驻留情况。我们提出AcceptMoE,一种验证器端专家选择器,结合目标路由得分与离线估计的承诺概率,自动调整每个验证块的可选专家数量,无需用户指定专家预算。在卸载时,AcceptMoE基于缓存驻留条件决定专家资格,而非预测自然路径并预取对应专家权重。尽管限制目标专家资格会改变模型分布,但在三个MoE目标和四个基准测试的12个模型-任务组合中,AcceptMoE的平均准确率仅比EAGLE-3推测解码低0.27个百分点。在批处理大小为1时,使用SGLang,其吞吐量达到基线的1.290倍(所有专家权重在GPU内存中),在物理专家卸载下提升至2.06倍,同时主机到设备的通信流量减少73.6%至77.1%。
原文摘要 · Abstract (English)
Speculative decoding verifies a tree of draft tokens in one target-model forward pass. For a mixture-of-experts (MoE) target, however, parallel verification can activate the union of the experts selected by all tree nodes, even though only a small subset of those nodes reaches the accepted output. Token count, activated-expert union size, and expert-weight traffic are therefore distinct cost measures: reducing the token workload need not shrink the expert union proportionally, and under offloading, transfer traffic also depends on cache residency. We introduce AcceptMoE, a verifier-side expert selector that combines target-router scores with offline-estimated commitment probabilities and automatically adjusts the number of eligible experts for each verification block, eliminating the need for a user-specified expert budget. Under offloading, AcceptMoE conditions expert eligibility on cache residency instead of predicting natural routes and prefetching the corresponding expert weights. Although constraining target-expert eligibility changes the model distribution, across 12 model-task pairs spanning three MoE targets and four benchmarks, AcceptMoE's mean accuracy is 0.27 percentage points lower than that of EAGLE-3 speculative decoding with natural routing. Served with SGLang at batch size one, it reaches 1.290 times the throughput of this baseline with all expert weights in GPU memory, and 2.06 times under physical expert offloading, while reducing host-to-device traffic by 73.6 percent to 77.1 percent.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。