通过专家预算控制，让MoE模型在并行推测解码时更高效。
MoE-Spec: Expert Budgeting for Efficient Speculative Decoding
- 推理时限制每层专家数量，只加载关键专家
- 相比现有方法提升10%~30%吞吐量
- 适合追求高吞吐、低延迟的部署场景
推测解码通过并行验证多个草案令牌加速大语言模型推理。然而，对于混合专家（MoE）模型，这种并行性带来严重瓶颈：庞大的草案树会激活大量唯一专家，显著增加内存压力，削弱推测解码相对于自回归解码的加速效果。现有方法在MoE验证代价过高时降低推测深度。我们提出MoE-Spec，一种无需训练的验证时专家预算方法，通过在每层设置固定专家容量上限，仅加载对验证贡献最大的专家，丢弃使用频率极低的尾部专家以减少带宽开销。在多个模型规模和数据集上的实验表明，该方法在保持相近质量的前提下，吞吐量比当前最优基线（EAGLE-3）高出10%~30%，且可通过收紧预算灵活权衡准确率与延迟。
原文摘要 · Abstract (English)
Speculative decoding accelerates Large Language Model (LLM) inference by verifying multiple drafted tokens in parallel. However, for Mixture-of-Experts (MoE) models, this parallelism introduces a severe bottleneck: large draft trees activate many unique experts, significantly increasing memory pressure and diminishing speedups from speculative decoding relative to autoregressive decoding. Prior methods reduce speculation depth when MoE verification becomes expensive. We propose MoE-Spec, a training-free verification-time expert budgeting method that decouples speculation depth from memory cost by enforcing a fixed expert capacity limit at each layer, loading only the experts that contribute most to verification and dropping the long tail of rarely used experts that drive bandwidth overhead. Experiments across multiple model scales and datasets show that this method yields 10--30\% higher throughput than state-of-the-art speculative decoding baselines (EAGLE-3) at comparable quality, with flexibility to trade accuracy for further latency reductions through tighter budgets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。