面向边缘设备的高效大模型推理加速器，兼顾低延迟与高能效。
Hybrid Systolic Array Accelerator with Optimized Dataflow for Edge Large Language Model Inference
- 采用混合数组架构与优化数据流，提升硬件利用率并减少内存访问。
- 在1.3B模型上实现247 token/s/mm²性能，较现有方案提升超2.45倍。
- 适合资源受限的边缘部署场景，支持长文本生成与高能效推理。
边缘大语言模型（LLM）推理提供安全、低延迟且成本可控的解决方案。本文强调，边缘加速器需在内存密集的解码阶段实现高面积效率和最小外部内存访问（EMA），同时在计算密集的预填充阶段保持高能效。提出一种基于混合系统阵列（HSA）架构的边缘LLM推理加速器，通过采用MXINT4权重量化并设计适配HSA的优化数据流，实现近乎零的反量化开销，在边缘DRAM带宽约束下达成100%硬件利用率，且精度损失极小。针对非线性运算，集成优化的均方根归一化（RMSNorm）与旋转位置编码（RoPE）单元，显著降低延迟、面积与内存开销，支持端到端推理。在运行1.3B模型的长输入/长输出场景中，性能达247/117（token/s/mm²），较现有方法提升超过2.45倍/13.5倍，同时在生成阶段保持优异能效。
原文摘要 · Abstract (English)
Edge inference for large language models (LLM) offers secure, low-latency, and cost-effective inference solutions. We emphasize that an edge accelerator should achieve high area efficiency and minimize external memory access (EMA) during the memory-bound decode stage, while maintaining high energy efficiency during the compute intensive prefill stage. This paper proposes an edge LLM inference accelerator featuring a hybrid systolic array (HSA) architecture that optimizes inference efficiency in both stages. To further reduce EMA, we adopt MXINT4 weight quantization and propose an optimized dataflow tailored for HSA, ensuring negligible dequantization overhead and achieving 100% hardware utilization with minimal accuracy loss under edge DRAM bandwidth constraints. For non-linear operations, we incorporate optimized root mean square normalization (RMSNorm) and rotary position embedding (RoPE) units, reducing their latency, area, and memory access overhead while enabling end-to-end inference on our accelerator. Our solution achieves 247/117 (token/s/mm2) while running a 1.3B LLM on long-input/long-output scenarios, providing >2.45x/13.5x improvement over existing approaches, while maintaining superior energy efficiency in token generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。