评测了移动端大模型量化效果，发现3.5比特/权重是性能关键阈值。
A Systematic Evaluation of On-Device LLMs: Quantization, Performance, and Resources
- 系统评估0.5B到14B模型在不同量化方法下的表现
- 3.5有效比特/权重是性能分水岭，超低精度模型反超高精度小模型
- 适合边缘设备部署的模型优化参考，尤其关注资源受限场景
将大语言模型（LLMs）部署在边缘设备上可提升隐私保护，但受限于资源而面临性能挑战。本文提出一种系统化评估方法，平衡模型能力、效率与资源约束。通过对0.5B至14B规模的模型及七种后训练量化（PTQ）方法在通用硬件上的广泛分析，我们发现：1）重度量化的大模型性能持续优于较小的高精度模型，性能阈值约为3.5有效比特/权重（BPW）；2）资源利用率随BPW线性增长，但功耗和内存占用因量化算法而异；3）随着模型缩小，吞吐量主要瓶颈从通信开销转为计算延迟。最后给出资源受限边缘环境中的模型优化建议。代码库见 https://anonymous.4open.science/r/LLMOnDevice/
原文摘要 · Abstract (English)
Deploying Large Language Models (LLMs) on edge devices enhances privacy but faces performance hurdles due to limited resources. We introduce a systematic methodology to evaluate on-device LLMs, balancing capability, efficiency, and resource constraints. Through an extensive analysis of models (0.5B-14B) and seven post-training quantization (PTQ) methods on commodity hardware, we demonstrate that: 1) Heavily quantized large models consistently outperform smaller, high-precision models, with a performance threshold at ~3.5 effective bits-per-weight (BPW); 2) Resource utilization scales linearly with BPW, though power and memory footprints vary by quantization algorithm; and 3) With a reduction in model size, the primary constraint on throughput transitions from communication overhead to computational latency. We conclude by offering guidelines for optimizing LLMs in resource-constrained edge environments. Our codebase is available at https://anonymous.4open.science/r/LLMOnDevice/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。