量化后模型行为可能已变，但传统指标仍显示效果一致。
The Illusion of Equivalency: Statistical Characterization of Quantization Effects in LLMs
- 提出新评估指标「正确性一致率」，看原始与量化模型预测重合度。
- 8到2比特量化下，模型行为普遍改变，即使准确率和困惑度不变。
- 查询与键投影比值和输出投影更易受量化影响，揭示行为差异根源。
后训练量化已成为压缩大语言模型以在资源受限设备上部署的常用方法。然而，现有量化评估主要依赖准确率和困惑度，无法捕捉量化模型的行为变化。本文提出「正确性一致率」这一决策级指标，用于衡量基础模型与其量化版本之间正确预测的交集。我们在多个模型和量化位宽（从8比特到2比特）上进行测试，发现即使准确率和困惑度保持不变，基础模型与量化版本的行为仍存在显著偏移。为解释此现象，我们通过统计与分布度量分析量化对注意力权重结构的影响，结果揭示低比特时存在断点，且查询与键投影比值投影和输出投影更敏感。这些发现证明了基础模型与量化模型之间的等效性是假象，呼吁采用超越困惑度和准确率的行为评估方法。
原文摘要 · Abstract (English)
Post-Training Quantization has become widely used to compress large language models to make them deployable on resource-constrained devices. However, the evaluation of quantization methods mainly uses accuracy and perplexity, which cannot capture the behavioral changes in the quantized variants. In this work, we propose Correctness Agreement, a decision-level metric that can measure the intersection of correct predictions between the base model and its quantized variant. We use this metric across multiple models and quantization bit levels (8-bit to 2-bit), and we find that the base and quantized variants usually have a shift in behavior even when accuracy and perplexity are preserved. In order to explain this effect, we study the effect of quantization on the structure of the attention weights using statistical and distributional measures. The results reveal a breakpoint at low bit widths and show that query and key projections are more sensitive to quantization than the value and output projections. These results prove the illusion of equivalency between the base and quantized models and inspire behavioral evaluation beyond perplexity and accuracy for quantization methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。