发现视觉语言模型推理链中的不确定性信号比答案更可靠，可提升判断准确性。
When Thinking Hurts: Epistemic Signals in the Reasoning Chains of Visual Language Models
- 通过分析三种模型的推理链熵，发现链信号比答案熵更能反映模型置信度。
- 在对抗样本上，推理链熵最高达0.759，显著优于答案熵的0.492至0.716。
- 提出无额外计算成本的弃权门控机制，准确率从71%提升至93.8%。
视觉语言模型（VLMs）的传统不确定性量化聚焦于答案词元分布。本文首次对思考模式下VLMs的答案熵行为进行三类实证分析。在相同POPE对抗样本上测试四款模型，发现三种截然不同的模式：Qwen3-VL-8B-Thinking出现完全坍塌（答案熵AUROC=0.492）；GLM-4.1V-9B-Thinking无坍塌（0.716）；InternVL3-8B仅在50%样本生成链（全量答案熵0.675，仅思考链0.602）。所有模型中，推理链熵在生成链的子集上均优于答案熵（0.647, 0.759, 0.608 vs. 0.492, 0.716, 0.602），表明链信号在存在时更具可靠性。该优势在Qwen和GLM中显著，在InternVL3中仅边际且统计不可靠（假阳性数n_FP=17）。300样本的VQAv2预实验验证链熵（0.680）优于答案熵（0.595），尤其在自由回答任务中差距最大（0.733 vs. 0.467）。在更难的推理任务（HallusionBench）中，两版Qwen模型均表现出中等信号（约0.64），与复杂问题上的不完全承诺一致。此外，观察到结构化弃权现象影响12-22%查询，且偏向缺失物体查询；设计实用弃权门控机制，在62.7%覆盖率下将准确率从71.0%提升至93.8%，无需额外推理开销。
原文摘要 · Abstract (English)
Uncertainty quantification for visual language models (VLMs) conventionally targets the answer token distribution. We provide the first three-family empirical characterisation of answer entropy behaviour in thinking-mode VLMs. Running four models on identical POPE adversarial samples, we find three qualitatively distinct patterns: Qwen3-VL-8B-Thinking shows complete collapse (ans H AUROC = 0.492); GLM-4.1V-9B-Thinking shows no collapse (0.716); and InternVL3-8B shows selective thinking (chains on only 50% of samples, ans H = 0.675 full / 0.602 thinking-only). Across all three thinking-mode models, thinking chain entropy outperforms answer entropy on the subset where chains are generated (0.647, 0.759, 0.608 vs. 0.492, 0.716, 0.602 respectively), suggesting chain signals are the more reliable predictor whenever chains are present. This holds strongly for Qwen and GLM, but with only marginal and statistically unreliable advantage for InternVL3 (n_FP = 17). A 300-sample VQAv2 pilot confirms chain entropy (0.680) outperforms answer entropy (0.595) on VQAv2 questions, with the gap largest for free-form answers (0.733 vs. 0.467). On harder reasoning tasks (HallusionBench) both Qwen models show moderate signal (approx. 0.64), consistent with incomplete pre-commitment on difficult questions. We additionally document structured abstention affecting 12-22% of queries with asymmetry toward absent-object queries, and a practical abstention gate raising accuracy from 71.0% to 93.8% at 62.7% coverage with no additional inference cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。