在手机上跑大模型，优化后的CPU竟比GPU还快。
Challenging GPU Dominance: When CPUs Outperform for On-Device LLM Inference
- 用llama.cpp在iPhone 15 Pro上部署10亿参数模型，仅用两个线程的CPU实现17词/秒
- 对比使用GPU加速的12.8词/秒，证明特定条件下CPU反超
- 适合关注移动端AI部署效率的开发者和研究者
当前主流观点认为，凭借更强的并行处理能力，GPU在移动端大语言模型（LLM）推理中始终占优。本文通过实证挑战这一认知：在特定条件下，CPU反而可超越GPU。我们在iPhone 15 Pro上使用llama.cpp部署一个10亿参数的LLM，采用双线程、半精度（F16）的纯CPU配置，达到17词/秒的推理速度，高于启用GPU加速时的12.8词/秒。我们分析了导致这一反直觉现象的架构因素，发现GPU内存传输开销和CPU线程优化起关键作用。此外，我们探讨了线程过度分配、量化策略及硬件限制的影响，为高效移动端AI执行提供了新洞见。尽管受限于iOS平台低级性能分析工具的缺失，仍难以完全解释观察到的CPU优势，但本研究挑战了以GPU为中心的传统思路，揭示了优化后CPU推理的巨大潜力，推动移动端智能部署策略的革新。
原文摘要 · Abstract (English)
The common assumption in on-device AI is that GPUs, with their superior parallel processing, always provide the best performance for large language model (LLM) inference. In this work, we challenge this notion by empirically demonstrating that, under certain conditions, CPUs can outperform GPUs for LLM inference on mobile devices. Using a 1-billion-parameter LLM deployed via llama.cpp on the iPhone 15 Pro, we show that a CPU-only configuration (two threads, F16 precision) achieves 17 tokens per second, surpassing the 12.8 tokens per second obtained with GPU acceleration. We analyze the architectural factors driving this counterintuitive result, revealing that GPU memory transfer overhead and CPU thread optimization play a critical role. Furthermore, we explore the impact of thread oversubscription, quantization strategies, and hardware constraints, providing new insights into efficient on-device AI execution. Our findings challenge conventional GPU-first thinking, highlighting the untapped potential of optimized CPU inference and paving the way for smarter deployment strategies in mobile AI. However, fully explaining the observed CPU advantage remains difficult due to limited access to low-level profiling tools on iOS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。