边端部署大模型,通过专家共享与推测解码降低内存访问开销。
EdgeXpert: An Edge Device for Memory-Efficient LLM Inference with Mixture-of-Experts and Speculative Decoding

- 按提示词重用专家,减少每轮推理的内存访问。
- 解码阶段仅加载关键专家通道,节省44.1%能耗。
- 适合资源受限的移动设备或边缘计算场景使用。
在设备端部署大型语言模型已成为个性化边缘应用的关键需求。主要瓶颈在于前馈网络层中的外部内存访问(EMA)。推测解码和混合专家(MoE)是两种有前景的解决方案:推测解码通过每轮生成多个标记减少解码轮次,而MoE通过稀疏专家激活降低每轮成本。然而两者存在兼容性问题。本文提出EdgeXpert,一种软硬件协同设计的LLM加速器,解决了该矛盾。预填充阶段采用提示词级专家复用,通过轻量编码器识别重要标记,构建共享专家集,并对次要标记使用更小的专家预算以降低专家级内存访问。解码阶段采用深度感知专家合并,利用同深度候选标记的上下文相似性与互斥性,不加载所有所需通道的并集,仅加载显著通道并通过计算校准恢复精度,无需额外内存访问。在三星28nm工艺下,工作频率为800 MHz,相比先前方案,最高实现56.3%延迟降低和44.1%能效提升,同时保持接近基线的准确率。
原文摘要 · Abstract (English)
On-device deployment of Large Language Models (LLMs) has become essential for personalized edge applications. A primary bottleneck is external memory access (EMA) in feed-forward network (FFN) layers. Speculative decoding and mixture-of-experts (MoE) are promising solutions. Speculative decoding reduces the number of decoding stages by generating multiple tokens per stage, and MoE minimizes per-stage cost through sparse expert activation. However, there is an incompatibility when combining these two techniques. We propose EdgeXpert, a software-hardware co-designed LLM accelerator that resolves this incompatibility. In the prefill stage, the prompt-wise expert reuse reformulates routing as prompt-level expert reuse rather than independent per-token expert selection. It identifies important tokens using a lightweight encoder, constructs a shared expert set from them, and routes less important tokens with a reduced expert budget to lower expert EMA. In the decode stage, depth-aware expert coalescing exploits the contextual similarity and mutual exclusivity of same-depth candidate tokens. Rather than loading the union of all required channels, EdgeXpert loads only salient channels and applies computational calibration to recover accuracy without additional memory access. Synthesized in Samsung 28nm technology at 800 MHz, EdgeXpert achieves up to 56.3% latency reduction and 44.1% energy reduction compared to prior works, while maintaining near-baseline accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。