在树莓派上实现低功耗高吞吐的大型语言模型推理
LLMPi: Optimizing LLMs for High-Throughput on Raspberry Pi
- 采用多种位宽的后训练量化技术，实现2-8比特权重压缩
- 结合量化解析训练使比特网络在低精度下保持准确率
- 为移动端和嵌入式设备部署大模型提供可行方案
在树莓派等资源受限的边缘设备上部署大型语言模型（LLMs）面临计算效率、功耗和响应延迟的挑战。本文探索基于量化优化的技术，以实现在低功耗嵌入式系统上的高吞吐、低功耗运行。方法包括k-量化（一种用于不同位宽的后训练量化技术），支持2比特、4比特、6比特和8比特权重量化；同时，针对BitNet模型采用量化解析训练（QAT）进行三值量化，更有效地适应低比特表示并保持准确性。结果表明，激进的量化策略可显著降低能耗，同时维持推理质量，使大型语言模型在资源受限环境中具备实际可行性，为移动与嵌入式应用中的低功耗高效人工智能部署开辟新路径。
原文摘要 · Abstract (English)
Deploying Large Language Models (LLMs) on resource-constrained edge devices like the Raspberry Pi presents challenges in computational efficiency, power consumption, and response latency. This paper explores quantization-based optimization techniques to enable high-throughput, energy-efficient execution of LLMs on low-power embedded systems. Our approach leverages k-quantization, a Post-Training Quantization (PTQ) method designed for different bit-widths, enabling efficient 2-bit, 4-bit, 6-bit, and 8-bit weight quantization. Additionally, we employ ternary quantization using Quantization-Aware Training (QAT) for BitNet models, allowing for more effective adaptation to lower-bit representations while preserving accuracy. Our findings highlight the potential of quantized LLMs for real-time conversational AI on edge devices, paving the way for low-power, high-efficiency AI deployment in mobile and embedded applications. This study demonstrates that aggressive quantization strategies can significantly reduce energy consumption while maintaining inference quality, making LLMs practical for resource-limited environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。