将视觉语言模型压缩至2.7比特，实现手机端高效推理
Llama-Mobile: Efficient 2.7-Bit Quantization of VLMs
- 用模型自生成数据完成量化，无需原始训练环境
- 110亿参数模型压缩至3.7GB，激活值保持8比特精度
- 适配移动端Arm CPU，适合资源受限设备部署
在移动设备上部署视觉语言模型（VLMs）面临内存与计算资源的双重挑战。本文提出一种针对VLMs的量化框架，实现资源受限硬件上的高效推理。该方法结合自生成训练数据的量化流程，无需访问原始训练设置；并引入创新的每参数2.7比特格式，支持在Arm CPU上高效执行。通过压缩Llama 3.2 11B Vision Instruct模型至3.7 GB，同时保持8比特激活精度，在多个标准视觉问答任务上维持优异性能。
原文摘要 · Abstract (English)
Deploying vision-language models (VLMs) on mobile devices is challenging due to their significant memory and compute requirements. We present a framework for quantizing VLMs for efficient inference on resource-constrained hardware. Our approach combines a quantization pipeline that uses the model itself to generate training data and does not require access to the training setup, with a novel 2.7-bit-per-parameter format supporting efficient execution on Arm CPUs. We validate our approach by compressing the Llama 3.2 11B Vision Instruct model to 3.7 GB with 8-bit activations, preserving strong performance on a set of standard visual question answering tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。