首次系统评估量化对长文本任务的影响，发现4比特压缩会大幅降性能。
Does quantization affect models' performance on long-context tasks?
- 测试9.7千例长文本任务，对比五种量化方法与五款大模型
- 8比特量化仅导致0.8%准确率下降，4比特最高损失达59%
- 跨语言和模型差异大，需针对性评估再部署
大型语言模型（LLMs）如今支持超过128K tokens的上下文窗口，但带来显著内存占用和高推理延迟。量化可缓解这些开销，但可能降低性能。本文首次系统评估量化后的LLM在长输入（>64K tokens）和长输出任务上的表现。评估涵盖9.7K测试样本、五种量化方法（FP8、GPTQ-int8、AWQ-int4、GPTQ-int4、BNB-nf4）及五款模型（Llama-3.1 8B和70B；Qwen-2.5 7B、32B、72B）。结果显示，平均而言，8比特量化仅造成约0.8%准确率下降，而4比特方法导致显著性能损失，尤其在长上下文输入任务中，降幅最高达59%。该退化在非英语输入时更严重。关键的是，量化效果高度依赖量化方法、模型和任务类型。例如，尽管Qwen-2.5 72B在BNB-nf4下保持稳健，但Llama-3.1 70B在同一任务中性能下降32%。这些发现强调，在长上下文场景及非英语任务中，部署量化模型前必须进行细致的任务特定评估。
原文摘要 · Abstract (English)
Large language models (LLMs) now support context windows exceeding 128K tokens, but this comes with significant memory requirements and high inference latency. Quantization can mitigate these costs, but may degrade performance. In this work, we present the first systematic evaluation of quantized LLMs on tasks with long inputs (>64K tokens) and long-form outputs. Our evaluation spans 9.7K test examples, five quantization methods (FP8, GPTQ-int8, AWQ-int4, GPTQ-int4, BNB-nf4), and five models (Llama-3.1 8B and 70B; Qwen-2.5 7B, 32B, and 72B). We find that, on average, 8-bit quantization preserves accuracy (~0.8% drop), whereas 4-bit methods lead to substantial losses, especially for tasks involving long-context inputs (drops of up to 59%). This degradation tends to worsen when the input is in a language other than English. Crucially, the effects of quantization depend heavily on the quantization method, model, and task. For instance, while Qwen-2.5 72B remains robust under BNB-nf4, Llama-3.1 70B experiences a 32% performance drop on the same task. These findings highlight the importance of a careful, task-specific evaluation before deploying quantized LLMs, particularly in long-context scenarios and for languages other than English.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。