让边缘设备上的专家模型更快推理，速度最高提升5.3倍。
S2-MoE: Enabling Efficient Self-Speculative Decoding for Mixture-of-Experts on Edge Devices
- 自适应推测生成，减少重复验证开销。
- 专家路由感知扩展，平均提速2.0倍，最高达5.3倍。
- 适合部署在内存受限的边缘设备，尤其适用MoE模型。
在边缘设备上部署大语言模型进行推理面临严重的内存和带宽限制。尽管推测解码与混合专家（MoE）模型已被提出以提升推理效率，但二者简单结合常导致过高的验证开销和专家复用率低，限制了其在内存受限边缘场景下的有效性。本文提出S2-MoE，一种面向边缘设备上MoE模型的高效自推测解码框架。通过路由感知的自适应推测扩展降低冗余验证，利用专家复用感知门控提升验证效率，并通过共享上下文对齐草稿与目标执行过程。在llama.cpp中实现，S2-MoE在多种MoE模型和数据集上，相较于标准自回归解码，最高提速5.3倍（平均约2.0倍）。代码已开源：https://github.com/angerybob/S2-MoE。
原文摘要 · Abstract (English)
Deploying large language models (LLMs) for inference on edge devices is challenging due to severe memory and bandwidth constraints. While speculative decoding and Mixture-of-Experts (MoE) have been proposed to improve inference efficiency, naively combining them often incurs excessive verification overhead and poor expert reuse, limiting their effectiveness in memory-bound edge settings. In this work, we propose S2-MoE, an efficient self-speculative decoding framework for MoE inference on edge devices. S2-MoE reduces redundant verification through routing-aware adaptive speculative expansion, improves verification efficiency with reuse-aware expert gating, and aligns draft and target execution via shared context. Implemented in llama$.$cpp, S2-MoE achieves up to $5.3\times$ speedup (about $2.0\times$ on average) over standard autoregressive decoding across diverse MoE models and datasets on edge devices. Code is available at https://github.com/angerybob/S2-MoE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。