量化压缩会悄悄改变模型答案背后的得分差距，但未必能提升性能。
BEFORE THE FLIP: Measuring Hidden Score Shifts In Quantized Vision Language Models Before The Answer Changes for Visual Question Answering
- 通过对比图像令牌替换与权重压缩对得分的影响，检测隐藏变化。
- 4位量化使答案得分差距更靠近平均令牌结果，8位量化影响较小。
- 按问题调整精度未优于随机控制，提示个性化压缩难有收益。
量化通过减少权重比特数降低视觉语言模型（VLM）的存储与运行成本。尽管压缩后视觉问答（VQA）答案不变是预期行为，但其背后得分（对数概率）可能已悄然改变。例如，模型仍回答“是”，但“是”与“否”的得分差距可能缩小。我们提出BEFORE THE FLIP方法，通过比较压缩引起的得分变化与用固定平均图像令牌替换原始图像表示所导致的变化，来探测这些隐藏差异。逐组提升权重精度以识别额外比特的增益，并测试为每个问题选择不同权重组是否优于随机对照。在8,277个LLaVA问题中，图像令牌替换显著影响得分；4位量化下，“是”或“否”的得分差距比8位量化更趋近于替换输出。Qwen呈现相同趋势，但差异较小。然而，在9,000个LLaVA答案中，仅265个在4位量化下发生改变。另一项针对1,024个校准问题的研究显示，按问题分别选择权重组并未在任何存储预算下优于随机对照。结果表明，压缩可改变不变答案背后的得分，但无法证明按问题动态调整精度具有可靠优势。
原文摘要 · Abstract (English)
Quantization makes vision language models (VLMs) cheaper to store and run by using fewer bits to represent their weights. While unchanged answers on visual question answering (VQA) after compression are an expected behavior, they can still hide changes in the underlying scores (log probabilities). For example, a model may still answer yes after compression, even as the score gap between yes and no shrinks. We introduce BEFORE THE FLIP to measure these hidden changes. Our method compares the score change caused by compression with the change caused by replacing the image's internal representations, or image tokens, with one fixed average token. We then increase the precision of one weight group at a time to identify where extra bits help, and test whether choosing different groups for each question offers benefits beyond shuffled controls. Among 8,277 LLaVA questions where image token replacement measurably affects the scores, 4-bit compression shifts the yes or no score gap farther toward the replacement output than 8-bit compression. Qwen shows the same pattern, but with a smaller difference. Yet only 265 of 9,000 LLaVA answers change at 4 bits. In a separate study of 1,024 calibration questions, choosing weight groups separately for each question does not outperform both shuffled controls at any tested storage budget. These findings show that compression can alter the scores behind unchanged answers, but do not establish a reliable benefit from adjusting precision for each question.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。