分析手机端大模型推理中NPU实际性能，发现计算密集阶段反被CPU更快。
When NPUs Are Not Always Faster: A Stage-Level Analysis of Mobile LLM Inference
- 通过分解执行管道，分离NPU中通信、量化与计算开销
- Prefill阶段CPU快1.6倍，Decode阶段NPU仅快1.05-1.2倍
- 调度开销和跨后端回退降低实际加速收益，还更耗电
在移动设备上部署大语言模型（LLMs）越来越依赖异构执行，但此前缺乏对NPU在操作符与流水线层级上的系统性评估。本文首次针对基于CPU-NPU异构SoC的移动端LLM推理，开展阶段感知、多层级基准测试。提出基于OPMASK的受控流水线分解方法，隔离了NPU执行路径中的通信、量化与计算开销。结果揭示反直觉的阶段级性能反转：在计算密集型Prefill阶段，CPU性能比NPU高至1.6倍；而在内存受限的Decode阶段，NPU仅带来1.05-1.2倍的加速。此外，调度开销与跨后端回退显著削弱了NPU卸载的实际效益。能效方面，增加NPU卸载反而导致能耗上升达51%。据此，我们为面向端侧LLM推理的NPU架构设计提供了指导原则。
原文摘要 · Abstract (English)
Deploying large language models (LLMs) on mobile devices increasingly relies on heterogeneous execution, yet no prior study has systematically characterized NPU effectiveness at the operator and pipeline level. We present the first stage-aware, multi-level benchmarking study of mobile LLM inference on a CPU-NPU heterogeneous SoC. We introduce an OPMASK-based controlled pipeline decomposition methodology that isolates communication, quantization, and computation overheads within the NPU execution path. Our results reveal a counter-intuitive stage-level performance reversal: CPUs outperform NPUs in the compute-intensive Prefill stage (up to 1.6x), while NPUs provide only limited acceleration in the memory-bound Decode stage (1.05-1.2x). We further show that scheduling overhead and cross-backend fallback reduce the practical benefits of NPU offloading. For the energy trend, increasing NPU offloading leads to higher energy consumption (up to 51%). Based on these findings, we derive design guidelines for NPU architects targeting on-device LLM inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。