对比芯片平台在边缘小模型推理中的表现,发现专用芯片最优。
Edge Deployment of Small Language Models, a comprehensive comparison of CPU, GPU and NPU backends
- 用统一框架测试主流CPU、GPU、NPU跑小模型的性能与功耗
- 专用NPU性能远超其他平台,能效比领先,适合边缘部署
- 低功耗ARM虽省电,但综合能效仍不如专用芯片,适合节能场景
边缘计算在数据生成处处理信息,实现更快决策、更低带宽和更好隐私。但边缘设备通常受限于算力、内存和能耗,难以运行大语言模型(LLMs)。小语言模型(SLMs)通过显著降低计算开销,成为资源受限环境下的轻量级替代方案。本文评估了商用CPU(Intel、ARM)、GPU(NVIDIA)和NPU(RaiderChip)在运行SLMs时的推理性能与能效。尽管GPU是常规选择,但本研究也对比了商业NPU与新型多核CPU。使用统一执行框架和一系列前沿SLMs,分析各平台的最大性能及处理与能效表现。结果表明,专用后端优于通用CPU,NPU以明显优势实现最高性能。带宽归一化对跨架构比较至关重要。虽然低功耗ARM处理器在能耗方面表现优异,但结合性能与功耗的指标(如EDP)再次凸显NPU主导地位。这说明为效率与性能优化的设计在边缘任务中具有显著优势。
原文摘要 · Abstract (English)
Edge computing processes data where it is generated, enabling faster decisions, lower bandwidth usage, and improved privacy. However, edge devices typically operate under strict constraints on processing power, memory, and energy consumption, making them unsuitable for large language models (LLMs). Fortunately, Small Language Models (SLMs) offer lightweight alternatives that bring AI inference to resource-constrained environments by significantly reducing computational cost while remaining suitable for specialization and customization. In this scenario, selecting the hardware platform that best balances performance and efficiency for SLM inference is challenging due to strict resource limitations. To address this issue, this study evaluates the inference performance and energy efficiency of commercial CPUs (Intel and ARM), GPUs (NVIDIA), and NPUs (RaiderChip) for running SLMs. GPUs, the usual platform of choice, are compared against commercial NPUs and recent multi-core CPUs. While NPUs leverage custom hardware designs optimized for computation, modern CPUs increasingly incorporate dedicated features targeting language-model workloads. Using a common execution framework and a suite of state-of-the-art SLMs, we analyze both maximum achievable performance and processing and energy efficiency across commercial solutions available for each platform. The results indicate that specialized backends outperform general-purpose CPUs, with NPUs achieving the highest performance by a wide margin. Bandwidth normalization proves essential for fair cross-architecture comparisons. Although low-power ARM processors deliver competitive results when energy usage is considered, metrics that combine performance and power (such as EDP) again highlight NPUs as the dominant architecture. These findings show that designs optimized for both efficiency and performance offer a clear advantage for edge workloads.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。