ThAME用三维异构架构解决大模型专家混合推理的内存与通信瓶颈。
ThAME: 3D Memory-Enabled Heterogeneous Accelerator for LLM Mixture of Experts
- 采用铁电晶体管非易失内存与动态内存协同设计，匹配不同计算需求。
- 在真实场景下实现最高15.7倍加速，能效提升9.8倍。
- 适合大规模语言模型推理系统研发者与硬件架构设计师参考。
专家混合（MoE）架构已成为扩展大型语言模型（LLM）的主流范式。然而，传统硬件上进行MoE推理受限于三大根本瓶颈：需频繁读取不连续的专家权重带来的巨大内存带宽压力、由输入依赖的令牌路由引发的非确定性散播-汇聚流量，以及同步专家输出聚合导致的尾延迟依赖。为此，我们提出ThAME，一种面向MoE推理的三维异构多芯片组架构。ThAME采用基于铁电场效应晶体管（FeFET）的非易失内存芯片组与基于动态随机存取存储器（DRAM）的易失内存芯片组，配合协同设计的计算映射策略，对齐注意力机制与专家路由的不同计算特性。此外，我们设计了专用片上网络通信骨干，以缓解输入依赖的MoE流量模式在组合空间中产生的非确定性路由流量瓶颈。实验结果表明，ThAME在速度上相比最先进方案最高提升15.7倍，在能效方面最高提升9.8倍。
原文摘要 · Abstract (English)
Mixture of Experts (MoE) architectures have emerged as a dominant paradigm for scaling Large Language Models (LLMs). However, MoE inference on conventional hardware is constrained by three fundamental bottlenecks. These encompass the massive memory bandwidth required to fetch non-contiguous expert weights, the non-deterministic scatter-gather traffic generated by input-dependent token routing, and the tail-latency dependency imposed by synchronous expert output aggregation. To address these challenges, we propose ThAME, a three-dimensional (3D) heterogeneous multi-chiplet architecture for MoE inference. ThAME employs Ferroelectric Field-Effect Transistor (FeFET)-based non-volatile and DRAM-based volatile memory chiplets with a co-designed compute mapping strategy that aligns the distinct computational profiles of attention mechanisms and expert routing. Furthermore, we design a specialized Network-on-Chip communication backbone optimized to mitigate the bottlenecks associated with non-deterministic token routing traffic across the combinatorial space of input-dependent MoE traffic patterns. Experimental results demonstrate that ThAME outperforms state-of-the-art counterparts by up to 15.7x in terms of speedup and improves energy efficiency by up to 9.8x.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。