新架构让大模型推理瓶颈从注意力转向专家分布与通信。
Rethinking LLM Inference Bottlenecks: Insights from Latent Attention and Mixture-of-Experts
- 用潜在注意力和专家混合提升计算强度，适配现代加速器
- 专家分摊到多设备上可平衡算力负载，避免性能瓶颈
- 适合关注硬件优化方向的研究者和系统开发者
传统Transformer模型的计算负载呈现显著二分：多头注意力（MHA）和分组查询注意力（GQA）因算术强度低而受内存限制，前馈网络则为计算密集型。这一差异长期推动专用硬件以缓解注意力瓶颈。本文指出，近年来的架构进步——多头潜在注意力（MLA）和专家混合（MoE）——引入了新的主导瓶颈，使问题重心从内存密集型注意力转向其他环节。首先，MLA的算术强度比MHA高出两个数量级以上，进入计算密集型范畴，更契合现代加速器。其次，将MoE专家分布于多个加速器上，通过批处理调节其算术强度至与稠密层相当，实现更均衡的计算分布。因此，硬件与系统优化的重点应从注意力加速转向高带宽互连及跨设备专家负载均衡。
原文摘要 · Abstract (English)
Computational workloads composing traditional transformer models are starkly bifurcated. Multi-Head Attention (MHA) and Grouped-Query Attention are memory-bound due to low arithmetic intensity, while FeedForward Networks are compute-bound. This dichotomy has long motivated research into specialized hardware to mitigate the attention bottleneck. This paper argues that recent architectural advances in transformer models -- Multi-head Latent Attention (MLA) and Mixture of Experts (MoE) -- introduce new dominant bottlenecks, shifting the challenge away from memory-intensive attention. We make two key observations. First, the arithmetic intensity of MLA is over two orders of magnitude higher than that of MHA, moving it toward a compute-bound regime well-matched to modern accelerators such as GPUs. Second, distributing MoE experts across a pool of accelerators allows batching to tune their arithmetic intensity to that of dense layers, producing a more balanced computational profile. Consequently, the focus of hardware and system optimization should shift from attention acceleration to high-bandwidth interconnects and balancing expert workloads across accelerators.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。