arXiv:2607.24440cs.CVcs.CL2026-07

大模型更准但自信心表达差,量化让自信信号暴跌

Bigger or Cheaper? Scale and Quantization Effects on Uncertainty Signals in Vision-Language Models Under Image Degradation

论文配图:Bigger or Cheaper? Scale and Quantization Effects on Uncertainty Signals in Vision-Language Models Under Image Degradation
图 1 · 摘自论文原文
  • 对比3种配置:小模型全精度、小模型4比特量化、大模型4比特量化
  • 7B模型内部不确定性信号AUROC达0.98,但口头自信仅0.61~0.69
  • 4比特量化几乎不降准确率(-1.6点),却使自信信号大幅下降

在六种真实图像退化场景下,针对Qwen2-VL系列模型,我们测量了模型规模与4比特量化对两种置信度信号的影响:模型自然语言声明的置信度,以及生成答案时的平均词概率。在5700次预测中发现,模型规模从2B增至7B,内部不确定性信号的错误检测AUROC从0.80提升至0.98,但口语化置信度始终弱且接近随机(0.61~0.69)。4比特量化对准确率影响极小(-1.6点),但严重损害置信信号:内部信号从0.95降至0.80,口头置信解析率从99%骤降至64%。因此,在固定内存预算下,应优先选择大模型4比特量化版本(7B-4bit),其同时具备最高准确率与最优不确定性信号(内部AUROC 0.98)。研究将结果定义为可直接部署的选择性预测操作点,并主张以误差检测AUROC而非校准误差作为区分两信号的关键指标。

原文摘要 · Abstract (English)

Vision-language models (VLMs) deployed on consumer hardware must decide when to answer and when to defer, and that decision depends on having a confidence signal that tracks correctness. A practitioner with a fixed memory budget faces a choice between a small model at full precision, the same small model quantized, and a larger model quantized into the same footprint -- three configurations that push the confidence signal in opposing directions. We measure, on identical inputs, how model scale and 4-bit quantization affect two confidence signals in the Qwen2-VL family: the confidence a model states in natural language, and its own mean token probability over the answer it generates. Across 5,700 predictions spanning six realistic photographic degradations at three severities, we find that scale sharply improves the model's internal uncertainty signal (mean error-detection AUROC 0.80 to 0.98 from 2B to 7B) while its verbalized confidence stays weak and often at chance (mean 0.61 to 0.69): the gap between what the model knows and what it says widens rather than closes with size. We find that 4-bit quantization is nearly free for accuracy (-1.6 points) but expensive for the confidence signal (internal AUROC 0.95 to 0.80, and the verbalized-confidence parse rate collapses from 99% to 64%). For a fixed memory budget the recommendation is therefore to prefer a larger quantized model over a smaller full-precision one: 7B-4bit gives both the best accuracy and the best uncertainty signal (internal AUROC 0.98) of the three configurations that fit. We frame the results as selective-prediction operating points so they translate directly into a deployment recommendation, and we argue that error-detection AUROC, not calibration error, is the metric that exposes the difference between the two signals.

视觉语言模型量化置信度模型规模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。