量化大模型时,准确率骗人,新方法能测出隐藏的输出分布失真。
Accuracy is Not Enough: A Divergence-Based Approach to Evaluate Fidelity Loss in Quantized LLMs

- 用分布差异度量量化前后模型输出变化,比传统准确率更敏感。
- 实验显示强量化下分布偏移明显上升,但准确率却可能不变或反升。
- 适合做边缘部署评估的工程师和研究人员参考,尤其关注模型可靠性者。
在内存受限的边缘设备上部署大语言模型(LLMs)依赖于激进的训练后量化。然而,当前评估主要基于零样本任务准确率,仅关注argmax预测结果,对预测分布的变化不敏感。这导致准确率在渐进量化过程中呈现不稳定、非单调行为,掩盖了与原始的BFloat16(BF16)基线模型相比的显著保真度损失,并给出误导性部署信号。本文提出一种基于分布的评估框架,通过计算全词表预测分布之间的差异(如Jensen-Shannon散度和总变差距离),量化量化后模型的信息损失。该方法可精细分析分布偏移。我们在五种基础架构和四个推理基准上进行120次实验,覆盖从未压缩的BF16到Q2_K的渐进量化。结果显示,发散度指标随量化强度增加而上升,补充了任务准确率提供的保真度信号。在llama.cpp方案中,混合精度的Q4_K在相似内存占用下,表现优于统一精度的Q4_0。这些发现表明,分布感知评估应作为任务准确率的实用补充诊断工具;它不直接衡量正确性、校准性、安全性或用户感知质量。
原文摘要 · Abstract (English)
Deployment of Large Language Models (LLMs) on memory-constrained edge devices relies heavily on aggressive post-training quantization. However, evaluating these models is largely based on zero-shot task accuracy, which depends solely on argmax predictions and is insensitive to changes in the underlying predictive distribution. Consequently, accuracy can exhibit unstable, non-monotonic behavior under progressive quantization, masking substantial fidelity loss relative to the BFloat16 (BF16) uncompressed base model and providing misleading deployment signals. We introduce a distribution-sensitive evaluation framework quantifying information loss in quantized LLMs as the divergence between full-vocabulary predictive distributions at the token decision boundary. We compute statistical distances, including Jensen-Shannon Divergence and Total Variation Distance, between outputs of full-precision and quantized models, enabling a fine-grained analysis of distributional shift. Using this framework, we quantify probability mass displacement and distributional drift relative to the BF16 reference, capturing predictive distribution changes not reflected in top-1 accuracy. We conduct a 120-run experimental matrix across five foundation architectures and four reasoning benchmarks under progressive quantization regimes, from uncompressed BF16 to Q2_K, providing a systematic fidelity analysis. Our results show divergence metrics generally increase under stronger quantization, complementing task accuracy with a fidelity signal. Across tested llama.cpp schemes, mixed-precision Q4_K generally yields lower divergence than uniform Q4_0 at similar memory footprints. These findings motivate distribution-aware evaluation as a practical diagnostic complement to task accuracy; they do not directly establish correctness, calibration, safety, or user-perceived quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。