首次揭示移动端大模型推理中NPU的隐藏能效瓶颈,提出优化配置可省电超5成。
Is Your NPU Ready for LLMs? Dissecting the Hidden Efficiency Bottlenecks in Mobile LLM Inference

- 跨框架跨硬件实测,用细粒度工具定位能效问题根源。
- 发现NPU在预填充阶段快10倍,但解码时反被CPU超越,因计算模式不匹配。
- 揭露调度漏洞,错误线程配置等导致最多40%能量浪费,适合移动设备开发者参考。
将大语言模型(LLMs)部署于移动端可提升隐私性并降低延迟,但受限于硬件效率低下。本文开展首个跨层、全方位的移动端LLM推理测量研究,覆盖五大主流框架(如llama.cpp、GENIE)及三类硬件后端(CPU、GPU、NPU)。为此,我们开发了PowerBench,首个支持后端级能耗归因的细粒度剖析工具,突破传统设备级测量局限。研究揭示三大关键洞见:(1) 框架差异在NPU上被显著放大,使用自定义算子时性能差距最高达10倍,源于离线策略与量化方案不同;(2) NPU在计算密集型预填充阶段表现优异,而内存密集型解码阶段则由CPU领先,因NPU偏好大且固定形状任务,与解码中小内核、动态特性冲突;(3) 后端特异性分析发现大量调度冗余,次优线程配置、未协调的NPU睡眠延迟及CPU轮询间隔导致高达40%的能量浪费。基于此,我们提出面向能耗优化的最佳实践配置,估算可在三个数据集上使NPU后端能耗降低最高54.8%。
原文摘要 · Abstract (English)
Deploying Large Language Models (LLMs) on mobile devices enhances privacy and reduces latency, but is severely bottlenecked by hardware inefficiency. We present the first comprehensive, cross-layer measurement study of mobile LLM inference, uniquely spanning five mainstream frameworks (e.g., llama.cpp, GENIE) and three hardware backends (CPU, GPU, NPU). To enable this analysis, we develop PowerBench, a fine-grained profiling tool that provides the first backend-specific energy attribution, moving beyond traditional device-level measurements. Our study yields three critical insights: (1) Framework-induced performance gaps are substantially amplified on NPUs, reaching up to 10x using custom operators due to divergent offloading and quantization strategies. (2) We identify a distinct phase split where NPUs excel at compute-bound prefilling, while CPUs outperform all other backends in memory-bound decoding. This is driven by the NPU's preference for large, fixed-shape workloads, which conflicts with the small-kernel, dynamic nature of decoding. (3) Backend-specific profiling uncovers substantial scheduling headroom missed by prior work. Suboptimal thread configurations, uncoordinated NPU sleep latencies, and CPU polling intervals result in up to 40% energy waste. Leveraging these findings, we present an energy-oriented best-practice configuration for mobile LLM inference. We estimate that this configuration could reduce energy consumption by up to 54.8% on the NPU backend across three datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。