优化专家模型推理速度,通过减少专家碎片化提升生成效率。
Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Experts

- 引入成本感知机制,选择能复用已有专家的生成路径
- 在多个千亿参数专家模型上实现最高1.62倍加速
- 适合追求高效推理的大型语言模型部署场景
稀疏专家混合(MoE)模型是扩展大语言模型的重要方法,但其推理效率高度依赖专家激活模式。推测解码(SD)通过并行验证多个候选词来加速自回归生成,但现有策略主要优化接受概率。在大规模MoE模型中,候选词选择还决定了验证过程中激活的专家集合。我们观察到,基于置信度的SD可能引发“专家分散”:高概率候选词可能分配给互不重叠的专家,增加专家权重内存访问量,降低推测加速效果。为此,我们重新审视MoE推理中非均匀内存开销下的候选树选择问题,提出EcoSpec——一种成本感知的推测解码框架,将预测的边际专家激活成本纳入候选选择。通过轻量级专家预测器与动态专家缓冲区,EcoSpec在不改变目标模型验证规则的前提下,优先选择高接受概率且能复用当前验证集已覆盖专家的路径。我们在三个大规模MoE模型(包括DeepSeek-V3.1 671B、Qwen3-235B-A22B和GPT-OSS-120B)上评估,涵盖推理、编程、问答和对话任务。EcoSpec持续减少活跃专家数量,提升端到端解码速度,最高达1.62倍加速。结果表明,在大规模MoE模型中考虑专家激活成本对高效推测解码至关重要。
原文摘要 · Abstract (English)
Sparse Mixture-of-Experts (MoE) models have become an important approach for scaling Large Language Models (LLMs), but their inference efficiency depends strongly on expert activation patterns. Speculative decoding (SD) accelerates autoregressive generation by verifying multiple draft tokens in parallel, yet existing draft selection strategies primarily optimize acceptance likelihood. In large-scale MoE models, however, selecting draft tokens also determines the union of experts activated during verification. We observe that confidence-driven SD can introduce \textit{expert scattering}: high-probability draft tokens may route to disjoint experts, increasing expert-weight memory traffic and reducing the speedup from speculation. Motivated by this observation, we revisit draft-tree selection under the non-uniform memory-cost structure of MoE inference. We propose \textsc{EcoSpec}, a cost-aware speculative decoding framework that incorporates predicted marginal expert activation cost into draft selection. With a lightweight expert predictor and a dynamic expert buffer, \textsc{EcoSpec} favors draft paths that preserve high acceptance likelihood while reusing experts already covered by the current verification set, without modifying the target-model verification rule. We evaluate \textsc{EcoSpec} on three large-scale MoE models, including DeepSeek-V3.1 (671B), Qwen3-235B-A22B, and GPT-OSS-120B, across reasoning, coding, question-answering, and dialogue benchmarks. \textsc{EcoSpec} consistently reduces active expert footprints and improves end-to-end decoding speed, achieving up to $1.62\times$ speedup. These results show that accounting for expert activation cost is important for efficient speculative decoding in large-scale MoE models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。