4-bit量化让大模型在手机上跑起来
Optimizing LLMs Using Quantization for Mobile Execution
- 用4比特后训练量化压缩模型，减少68.66%体积
- 量化后模型可在安卓设备成功运行并完成推理
- 适合想在移动端部署大模型的开发者参考
大型语言模型（LLMs）功能强大，但其庞大的规模和计算需求限制了在资源受限的移动设备上的部署。本文研究了用于移动执行的后训练量化（PTQ）。我们使用BitsAndBytes库与Hugging Face Transformers框架，对Meta的Llama 3.2 3B模型实施4比特PTQ。通过llama.cpp工具将量化模型转换为GGUF格式，实现移动端优化推理。该流程使模型尺寸减少68.66%，使Llama 3.2 3B模型能在安卓设备上高效运行。定性验证表明，4比特量化模型可成功完成推理任务。我们展示了在安卓设备上通过Termux环境与Ollama框架运行量化后的GGUF模型的可行性。4比特精度结合如GGUF等移动端优化格式的PTQ，为在移动设备上部署高性能大模型提供了实用路径，平衡了模型大小与性能。
原文摘要 · Abstract (English)
Large Language Models (LLMs) offer powerful capabilities, but their significant size and computational requirements hinder deployment on resource-constrained mobile devices. This paper investigates Post-Training Quantization (PTQ) for compressing LLMs for mobile execution. We apply 4-bit PTQ using the BitsAndBytes library with the Hugging Face Transformers framework to Meta's Llama 3.2 3B model. The quantized model is converted to GGUF format using llama.cpp tools for optimized mobile inference. The PTQ workflow achieves a 68.66% reduction in model size through 4-bit quantization, enabling the Llama 3.2 3B model to run efficiently on an Android device. Qualitative validation shows that the 4-bit quantized model can perform inference tasks successfully. We demonstrate the feasibility of running the quantized GGUF model on an Android device using the Termux environment and the Ollama framework. PTQ, especially at 4-bit precision combined with mobile-optimized formats like GGUF, provides a practical pathway for deploying capable LLMs on mobile devices, balancing model size and performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。