对比多种量化方案，帮你在本地运行Llama-3时选对合适设置。
Which Quantization Should I Use? A Unified Evaluation of llama.cpp Quantization on Llama-3.1-8B-Instruct
- 统一评测Llama-3.1-8B-Instruct在llama.cpp中的3-8比特及旧格式量化
- 覆盖推理、知识、指令遵循等任务表现与速度、压缩率等指标
- 适合关注本地部署效率的开发者和研究者参考
量化是降低大语言模型存储与计算精度以提升部署可行性的实用技术，可减少内存占用并在资源受限设备上实现高效运行，尤其适用于本地运行场景。llama.cpp 中的量化使大模型能在消费级硬件上运行，但现有格式评估标准不一，难以选择。本文针对单个现代模型 Llama-3.1-8B-Instruct（FP16, GGUF）开展统一实证研究，涵盖 3-8 位 K-量化及传统格式。评估内容包括标准推理、知识、指令遵循与真实性基准下的下游任务性能，同时测量困惑度、CPU 吞吐率（预填充/解码）、模型大小、压缩率与量化耗时。本研究为选择 llama.cpp 量化方案提供实用指南，帮助用户根据使用场景与资源预算做出知情决策。
原文摘要 · Abstract (English)
Quantization is a practical technique for making large language models easier to deploy by reducing the precision used to store and operate on model weights. This can lower memory use and improve runtime feasibility on constrained hardware, which is especially relevant for users running models locally. Quantization in llama.cpp enables large language models to run on commodity hardware, but available formats are often evaluated inconsistently, making it hard to choose among schemes. We present a unified empirical study of the llama.cpp quantization on a single modern model, Llama-3.1-8B-Instruct (FP16, GGUF), covering 3-8 bit K-quant and legacy formats. We evaluate downstream task performance across standard reasoning, knowledge, instruction-following, and truthfulness benchmarks, and also measure perplexity and CPU throughput (prefill/decoding) alongside model size, compression, and quantization time. Ultimately, this work is a practical guide for choosing a llama.cpp quantization scheme, helping readers make informed, context-aware decisions for their intended use and resource budget.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。