量化大模型时,传统精度指标在接近基线时失效,需警惕误判。
Displacement Is Not Direction: Evaluating Fidelity Metrics for Quantized LLM Deployment
- 用词级KL散度评估量化模型质量,但仅在显著偏离基线时有效。
- 在接近基线的静默区,指标与实际性能相关性消失,相关系数趋近于零。
- 指标主要反映差异量而非方向,不适合用于跨模型性能预测或故障预警。
Fidelity metrics(如与高精度参考模型的逐词KL散度)常被用作低代价的质量代理指标。我们在Qwen3.6-35B-A3B的28种量化方案和Devstral-Small-2-24B的41种量化方案上,通过多个下游基准测试验证该做法。结果发现,在全量程范围内,KL散度与基准得分强相关(Qwen: ρ = -0.72, p < 0.001;Devstral: ρ = -0.86, p < 0.001),但在接近基线的静默区域,相关性完全丧失(Qwen: ρ = +0.00;Devstral: ρ = -0.24, p = 0.36)。此现象在14种测量变体中均持续存在,包括不同聚合方式、困惑度形式、top-1一致率、校准语料和上下文长度。在单个提示层面,KL散度对代码任务的失败预测能力弱,失败与通过样本的几何均值比在[1.08,1.22]之间,且无法作为跨模型路由工具,对分歧提示的准确率仅为42.3%–49.4%。我们溯源发现,KL散度主要衡量与参考模型的差异体积,其在静默区的复合相关系数分别为Qwen: +0.94 (p < 0.001),Devstral: +0.55 (p = 0.03),而与差异方向的关系较弱且依赖任务。
原文摘要 · Abstract (English)
Fidelity metrics, such as per-token KL divergence (KLD) against a high-precision reference, are often used in practice as low-cost proxies for benchmark quality. We test this practice on a 28-quant cohort of Qwen3.6-35B-A3B and a 41-quant cohort of Devstral-Small-2-24B, evaluated across a suite of downstream benchmarks. We find that KLD is strongly correlated with benchmark score over the full cohort ($ρ=-0.72$ on Qwen and $ρ=-0.86$ on Devstral, both with $p<0.001$). However, this relationship collapses to non-significance in the near-baseline silent zone ($ρ=+0.00$ on Qwen and $ρ=-0.24$, $p=0.36$, on Devstral). This collapse persists across 14 measurement variants, including different KLD aggregations, perplexity formulations, top-1 agreement, calibration corpora, and context lengths. At the per-prompt level, KLD has only weak failure-prediction power on code, with failed-vs-passed geometric-mean ratios in $[1.08,1.22]$ across five models on LiveCodeBench, and fails as a cross-model router, achieving only $42.3\%-49.4\%$ accuracy on disagreement prompts. We trace the collapse to a structural decomposition: KLD primarily measures the volume of disagreement with the reference, with silent-zone composite $ρ=+0.94$ ($p<0.001$) on Qwen and $+0.55$ ($p=0.03$) on Devstral, while its relationship to the direction of those disagreements is weak and task-conditional.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。