对比四类设备在持续运行下大模型推理的能效表现，揭示硬件约束差异。
LLM Inference at the Edge: Mobile, NPU, and GPU Performance Efficiency Trade-offs Under Sustained Load
- 在固定提示下测试四类设备的持续推理性能，关注功耗与温控影响。
- 手机因过热导致吞吐量下降近一半，部分设备被系统强制降频中断任务。
- 专用芯片在低功耗下稳定运行，能效比优于移动端和笔记本显卡。
将大型语言模型部署于本地以支持始终在线的个人助手，需在功耗、散热和内存受限的硬件上实现持续推理。本文对4位量化后的Qwen 2.5 1.5B模型在四种平台上的表现进行基准测试：搭载Hailo-10H NPU的Raspberry Pi 5、三星Galaxy S24 Ultra、iPhone 16 Pro以及配备NVIDIA RTX 4050 GPU的笔记本电脑。采用固定258词提示，在每台设备上执行20次预热迭代后测量吞吐量、延迟、功耗及热行为。移动平台中，热管理成为主要瓶颈：iPhone 16 Pro在两次迭代内吞吐量几乎减半，S24 Ultra因操作系统强制设置GPU频率下限而完全终止推理。专用硬件则受不同因素制约：RTX 4050受限于电池供电上限，而Hailo-10H受限于模块内内存带宽。其中，RTX 4050在34.1 W下维持131.7 tok/s；Hailo-10H在低于2 W功耗下稳定输出6.9 tok/s，波动极小，且能效比例与RTX 4050相当，但吞吐量仅为后者的1/19。结果反映特定模型与提示类型下平台级部署特性，体现软硬件协同效应，非单纯硬件能力评价。
原文摘要 · Abstract (English)
Deploying large language models on-device for always-on personal agents demands sustained inference from hardware tightly constrained in power, thermal envelope, and memory. We benchmark Qwen 2.5 1.5B (4-bit quantised) across four platforms: a Raspberry Pi 5 with Hailo-10H NPU, a Samsung Galaxy S24 Ultra, an iPhone 16 Pro, and a laptop NVIDIA RTX 4050 GPU. Using a fixed 258-token prompt over 20 warm-condition iterations per device, we measure throughput, latency, power, and thermal behaviour. For mobile platforms, thermal management supersedes peak compute as the primary constraint: the iPhone 16 Pro loses nearly half its throughput within two iterations, and the S24 Ultra suffers a hard OS-enforced GPU frequency floor that terminates inference entirely. On dedicated hardware, distinct constraints dominate: the RTX 4050 is bounded by its battery power ceiling, while the Hailo-10H is limited by on-module memory bandwidth. The RTX 4050 sustains 131.7 tok/s at 34.1 W; the Hailo-10H sustains 6.9 tok/s at under 2 W with near-zero variance, matching the RTX 4050 in energy proportionality at 19x lower throughput. Results should be interpreted as platform-level deployment characterisations for a single model and prompt type, reflecting hardware and software combined, rather than general claims about hardware capability alone.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。