通过块对角稀疏性优化，实现内存内计算加速大模型推理。
Efficient In-Memory Acceleration of Sparse Block Diagonal LLMs
- 利用块对角稀疏结构设计新映射与调度策略。
- 内存占用和浮点运算量减少超4倍，阵列利用率提升50%以上。
- 适合资源受限设备上部署高效大模型推理的场景。
结构化稀疏性使大语言模型（LLMs）能在资源受限系统上部署。如密集转稀疏微调方法，可将模型规模压缩超过6.7倍，同时保持可接受的准确率。尽管如此，尤其是解码阶段固有的内存瓶颈，在传统冯·诺依曼架构下仍导致推理成本极高。计算-内存（CIM）架构通过在内存中直接计算，缓解此问题；结合稀疏LLMs后，可将整个模型存于内存，消除片外总线数据搬运，显著提升效率。然而，将稀疏矩阵简单映射至CIM阵列会导致阵列利用率低，计算效率下降。本文提出自动化框架，结合新颖的映射与调度策略，加速稀疏LLM在CIM加速器上的推理。通过利用块对角稀疏性，该方法使CIM阵列利用率提升超过50%，实现内存占用和所需浮点运算量均降低4倍以上。
原文摘要 · Abstract (English)
Structured sparsity enables deploying large language models (LLMs) on resource-constrained systems. Approaches like dense-to-sparse fine-tuning are particularly compelling, achieving remarkable structured sparsity by reducing the model size by over 6.7x, while still maintaining acceptable accuracy. Despite this reduction, LLM inference, especially the decode stage being inherently memory-bound, is extremely expensive on conventional Von-Neumann architectures. Compute-in-memory (CIM) architectures mitigate this by performing computations directly in memory, and when paired with sparse LLMs, enable storing and computing the entire model in memory, eliminating the data movement on the off-chip bus and improving efficiency. Nonetheless, naively mapping sparse matrices onto CIM arrays leads to poor array utilization and diminished computational efficiency. In this paper, we present an automated framework with novel mapping and scheduling strategies to accelerate sparse LLM inference on CIM accelerators. By exploiting block-diagonal sparsity, our approach improves CIM array utilization by over 50%, achieving more than 4x reduction in both memory footprint and the number of required floating-point operations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。