评测25个量化模型在三款单板机上的推理表现，揭示边缘部署可行性。
An Evaluation of LLMs Inference on Popular Single-board Computers
- 在Raspberry Pi等单板机上运行25个量化LLM，对比Ollama与Llamafile性能。
- 1.5B参数以下模型可在单板机稳定运行，Llamafile吞吐量提升4倍，功耗降30%-40%。
- 首次系统评估边缘设备上LLM推理，适合关注本地化、低功耗AI部署的研究者。
随着对设备端大语言模型（LLM）推理需求的增长，轻量级、低成本的AI解决方案在边缘硬件上的部署日益受到关注。树莓派和橙色派等单板计算机（SBC）为本地化、隐私保护的推理提供了有前景的平台，但在LLM工作负载方面的研究仍不充分。本文在三种SBC（Raspberry Pi 4、Raspberry Pi 5、Orange Pi 5 Pro）上，对25个量化开源LLM进行了基准测试，使用Ollama和Llamafile两种推理运行时，评估生成吞吐量、内存占用和功耗，采用多种提示类型模拟真实负载。结果表明，单板机可稳定支持最高1.5B参数的模型，且Llamafile相比Ollama实现最高4倍吞吐量提升，功耗降低30%-40%。我们识别出架构相关瓶颈，揭示运行时层面的权衡，并提供实用部署建议。本研究首次全面评估了SBC上的LLM推理，弥合高性能语言模型与经济型边缘计算之间的差距。
原文摘要 · Abstract (English)
The growing demand for on-device large language model (LLM) inference is driving interest in deploying lightweight, cost-effective AI solutions on edge hardware. Single-board computers (SBCs) such as the Raspberry Pi and Orange Pi offer a promising platform for localized, privacy-preserving inference-but remain underexplored in the context of LLM workloads. In this work, we benchmark the performance of 25 quantized open-source LLMs across three SBCs-Raspberry Pi 4, Raspberry Pi 5, and Orange Pi 5 Pro-using two inference runtimes: Ollama and Llamafile. We evaluate generation throughput, memory usage, and power consumption under varying CPU configurations, using multiple prompt types to simulate realistic workloads. Our results show that SBCs can reliably support models up to 1.5B parameters, with Llamafile achieving up to 4x higher throughput and 30-40% lower power usage than Ollama. We identify architecture-specific bottlenecks, highlight runtime-level trade-offs, and provide practical deployment recommendations. This study offers the first broad evaluation of LLM inference on SBCs, bridging the gap between high-performance language models and affordable edge computing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。