arXiv:2607.24434cs.LGcs.AI2026-07

让手机端大模型推理更快,通过智能预判减少专家加载延迟。

DraftExpert: Expansion-Aware Self-Speculative Decoding for End-Device MoE Inference

论文配图:DraftExpert: Expansion-Aware Self-Speculative Decoding for End-Device MoE Inference
图 1 · 摘自论文原文
  • 用轻量级专家模拟主模型,动态预测最可能激活的专家。
  • 在不同设备间切换时,吞吐提升1.45倍,预判准确率达84%~87%。
  • 适合移动端或资源受限设备部署大模型,尤其对实时性要求高场景。

大型混合专家(MoE)语言模型因其每标记仅激活少量专家而适合终端设备部署,但路由专家权重常超出加速器内存。我们针对延迟敏感的单用户场景,将激活专家按需从CPU内存加载至GPU,或从闪存加载至移动NPU。在此场景下,自推测解码面临新瓶颈:扩大草稿专家集可提高准确性,但触发额外专家加载;而低成本小规模草稿接受率低;验证多标记块会激活目标专家的并集,不再接近单步目标。我们提出DraftExpert,一种面向专家卸载式MoE推理的扩增感知自推测解码框架。DraftExpert通过自蒸馏冻结目标MoE的残差、逻辑值、标记及路由器一致性信号,在每层训练一个轻量级驻留于加速器的草稿专家。推理时,采用固定大小的共享+top-1+草稿专家生成器,结合置信度-扩增截断与目标专家预取策略,最终标记仍由目标模型精确验证。在DeepSeek-V2-Lite和Moonlight-16B-A3B上,跨CPU-GPU与Flash-NPU卸载,DraftExpert平均提升解码吞吐1.45倍,草稿接受率提升至84%~87%,预取命中率达86%~88%。

原文摘要 · Abstract (English)

Large Mixture-of-Experts (MoE) language models are attractive for end-device deployment because only a small subset of experts is active per token, but their routed expert weights often exceed accelerator memory. We target latency-critical single-user settings where routed experts are staged on demand from CPU memory to a GPU or from Flash to a mobile NPU. In this setting, self-speculative decoding faces a new bottleneck: increasing the draft expert set improves accuracy but triggers extra expert loading, while cheap small-footprint drafts have low acceptance; moreover, verifying a multi-token block activates the union of target experts and is no longer close to one target step. We propose DraftExpert, an expansion-aware self-speculative decoding framework for expert-offloaded MoE inference. DraftExpert trains one lightweight accelerator-resident draft expert per layer by self-distilling residual, logit/token, and router-agreement signals from the frozen target MoE. At inference time, it uses a fixed-footprint shared+top-1+draft-expert drafter together with confidence--expansion truncation and target-expert prefetching, while final tokens are still exactly verified by the target model. On DeepSeek-V2-Lite and Moonlight-16B-A3B across CPU-GPU and Flash-NPU offload, DraftExpert improves decode throughput by 1.45x on average, raises draft acceptance to 84~87%, and achieves 86~88% prefetch hit rates.

MoE推理自推测移动端优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。