arXiv:2604.14626cs.LGcs.AI2026-04

通过硬件协同设计,让MoE模型在本地部署时更省内存、更快推理。

ELMoE-3D: Leveraging Intrinsic Elasticity of MoE for Hybrid-Bonding-Enabled Self-Speculative Decoding in On-Premises Serving

论文配图:ELMoE-3D: Leveraging Intrinsic Elasticity of MoE for Hybrid-Bonding-Enabled Self-Speculative Decoding in On-Premises Serving
图 1 · 摘自论文原文
  • 利用混合键合技术与专家缓存,实现自预测解码的弹性扩展
  • 在批量大小1-16下平均提速6.6倍,能效提升4.4倍
  • 适合需要高效本地部署MoE模型的系统设计者和工程师

混合专家(MoE)模型已成为大规模语言模型的主流架构,但本地部署仍受内存限制,因批处理将稀疏的每标记计算转为密集的内存激活。以内存为中心的架构(如PIM、NMP)虽提升带宽,但在高批量下因计算强度低导致算力浪费。自预测解码(SD)可将闲置算力转化为减少目标调用次数,但验证阶段仍需加载专家,尤其在低批量下收益受限。我们提出ELMoE-3D,一种基于混合键合(HB)的软硬件协同框架,融合缓存加速与自预测解码,在全批量范围内实现整体加速。我们发现MoE专家与比特粒度的内在弹性特性,联合缩放构建弹性自预测解码(Elastic-SD),既充当专家缓存,又作为由高HB带宽加速的强对齐自草稿模型。基于比特切片的低位增强架构利用比特切片表示中的固有冗余,原生支持比特嵌套执行。在3D堆叠硬件上,ELMoE-3D相比朴素MoE服务在批量1-16范围内平均提速6.6倍,能效提升4.4倍;相比最优前代加速器基线,提速2.2倍,能效提升1.4倍。

原文摘要 · Abstract (English)

Mixture-of-Experts (MoE) models have become the dominant architecture for large-scale language models, yet on-premises serving remains fundamentally memory-bound as batching turns sparse per-token compute into dense memory activation. Memory-centric architectures (PIM, NMP) improve bandwidth but leave compute underutilized under MoE's low arithmetic intensity at high batch sizes. Speculative decoding (SD) trades idle compute for fewer target invocations, yet verification must load experts even for rejected tokens, severely limiting its benefit in MoE especially at low batch sizes. We propose ELMoE-3D, a hybrid-bonding (HB)-based HW-SW co-designed framework that unifies cache-based acceleration and speculative decoding to offer overall speedup across batch sizes. We identify two intrinsic elasticity axes of MoE-expert and bit-and jointly scale them to construct Elastic Self-Speculative Decoding (Elastic-SD), which serves as both an expert cache and a strongly aligned self-draft model accelerated by high HB bandwidth. Our LSB-augmented bit-sliced architecture exploits inherent redundancy in bit-slice representations to natively support bit-nested execution. On our 3D-stacked hardware, ELMoE-3D achieves an average $6.6\times$ speedup and $4.4\times$ energy efficiency gain over naive MoE serving on xPU across batch sizes 1-16, and delivers $2.2\times$ speedup and $1.4\times$ energy efficiency gain over the best-performing prior accelerator baseline.

MoE推理加速硬件协同节能

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。