4-bit量化让小模型更省资源，性能损失可控。
LLM Compression: How Far Can We Go in Balancing Size and Performance?
- 用4位分组缩放和GPTQ对三款小模型压缩
- 在多个任务上保持高准确率，延迟降低超60%
- 适合部署在资源受限设备的开发者参考
量化是提升大语言模型可访问性的关键技术，通过降低内存占用和计算成本来维持性能。本研究将4比特分组缩放量化（GSQ）和生成式预训练变换器量化（GPTQ）应用于LLaMA 1B、Qwen 0.5B和PHI 1.5B模型，在MS MARCO（信息检索）、BoolQ（布尔问答）和GSM8K（数学推理）数据集上评估其表现。对比了压缩后模型在准确率、推理延迟和吞吐量（每秒生成总词数）上的权衡，揭示低比特量化在真实场景部署中的适用性。分析不同规模模型在两种量化方法下的优劣，为后续实验提供基准。
原文摘要 · Abstract (English)
Quantization is an essential and popular technique for improving the accessibility of large language models (LLMs) by reducing memory usage and computational costs while maintaining performance. In this study, we apply 4-bit Group Scaling Quantization (GSQ) and Generative Pretrained Transformer Quantization (GPTQ) to LLaMA 1B, Qwen 0.5B, and PHI 1.5B, evaluating their impact across multiple NLP tasks. We benchmark these models on MS MARCO (Information Retrieval), BoolQ (Boolean Question Answering), and GSM8K (Mathematical Reasoning) datasets, assessing both accuracy and efficiency across various tasks. The study measures the trade-offs between model compression and task performance, analyzing key evaluation metrics, namely accuracy, inference latency, and throughput (total output tokens generated per second), providing insights into the suitability of low-bit quantization for real-world deployment. Using the results, users can then make suitable decisions based on the specifications that need to be met. We discuss the pros and cons of GSQ and GPTQ techniques on models of different sizes, which also serve as a benchmark for future experiments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。