针对边缘设备小模型推理，提出软硬件协同设计的存内计算方案。
EdgeCIM: A Hardware-Software Co-Design for CIM-Based Acceleration of Small Language Models
- 采用存内计算宏与分块映射策略，平衡流水线并缓解内存瓶颈。
- 在4B参数模型上实现平均336.42 tokens/s和173.02 tokens/J能效。
- 相比Orin Nano和SA8255P，吞吐提升达7.3倍与9.95倍，适合边缘部署。
随着小语言模型（SLMs）在笔记本、手机及嵌入式平台上的部署需求增长，现有加速器暴露出根本性效率问题。尽管GPU能高效处理预填充任务，但自回归解码阶段以GEMV运算为主，本质为内存密集型，导致边缘端利用率低且能耗过高。本文提出EdgeCIM，一种面向端到端仅解码器推理的软硬件协同设计框架。核心为65nm工艺实现的存内计算（CIM）宏，结合基于区块的映射策略，平衡流水线阶段，最大化并行性并缓解DRAM带宽瓶颈。通过仿真器对最大4B参数的SLMs进行设计空间探索，识别出延迟与能耗的帕累托最优配置。相较于NVIDIA Orin Nano，EdgeCIM在LLaMA3.2-1B上实现最高7.3倍吞吐量与49.59倍能效提升；在LLaMA3.2-3B上相较Qualcomm SA8255P达9.95倍吞吐提升。在TinyLLaMA-1.1B、LLaMA3.2（1B, 3B）、Phi-3.5-mini-3.8B、Qwen2.5（0.5B, 1.5B, 3B）、SmolLM2-1.7B、SmolLM3-3B及Qwen3（0.6B, 1.7B, 4B）等模型上的广泛测试表明，于INT4精度下，平均达336.42 tokens/s吞吐与173.02 tokens/J能效，确立其作为实时、低功耗边缘推理的有效方案。
原文摘要 · Abstract (English)
The growing demand for deploying Small Language Models (SLMs) on edge devices, including laptops, smartphones, and embedded platforms, has exposed fundamental inefficiencies in existing accelerators. While GPUs handle prefill workloads efficiently, the autoregressive decoding phase is dominated by GEMV operations that are inherently memory-bound, resulting in poor utilization and prohibitive energy costs at the edge. In this work, we present EdgeCIM, a hardware-software co-design framework that rethinks accelerator design for end-to-end decoder-only inference. At its core is a CIM macro, implemented in 65nm, coupled with a tile-based mapping strategy that balances pipeline stages, maximizing parallelism while alleviating DRAM bandwidth bottlenecks. Our simulator enables design space exploration of SLMs up to 4B parameters, identifying Pareto-optimal configurations in terms of latency and energy. Compared to an NVIDIA Orin Nano, EdgeCIM achieves up to 7.3x higher throughput and 49.59x better energy efficiency on LLaMA3.2-1B, and delivers 9.95x higher throughput than Qualcomm SA8255P on LLaMA3.2-3B. Extensive benchmarks on TinyLLaMA-1.1B, LLaMA3.2 (1B, 3B), Phi-3.5-mini-3.8B, Qwen2.5 (0.5B, 1.5B, 3B), SmolLM2-1.7B, SmolLM3-3B, and Qwen3 (0.6B, 1.7B, 4B) reveal that our accelerator, under INT4 precision, achieves on average 336.42 tokens/s and 173.02 tokens/J. These results establish EdgeCIM as a compelling solution towards real-time, energy-efficient edge-scale SLM inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。