检验视觉语言模型的解码选择机制，发现扰动一致性并非有效信号。
It's the Decoding Format, Not the Perturbation: Auditing Consistency-Based Selection for Vision-Language Test-Time Scaling
- 用输入扰动检测答案一致性，作为选择候选答案的标准
- 在多个数据集上，扰动方法比纯多数投票最高提升31.8分
- 但控制解码格式后，该方法优势消失，不具实际使用价值
测试时扩展通过采样多个候选解并筛选来提升大模型推理能力，但此方法在视觉语言模型（VLMs）中表现不佳：已有研究表明，简单多数投票优于基于模型自验证的筛选方法，因图像相关答案与语言先验的自信猜测在筛选层难以区分。自然对策是使筛选信号依赖于图像。本文研究无标签、无需训练的扰动一致筛选（Pgs）：通过在保持标签的前提下对输入进行裁剪、背景遮蔽、轻微光照或几何扰动，判断模型是否重新推导出相同答案；当扰动集为空时，Pgs退化为多数投票。关键问题是：在控制解码格式和预算的前提下，扰动项是否仍有效？为此引入格式匹配对照组（MatchedCtrl）——在原图上使用相同短文本、无链式思维的抽样策略。在TextVQA、MATH-Vision、MMMU和ViLP上，以Qwen为主干模型（三种子实验均值），且覆盖LLaVA-OneVision，在预算匹配的筛选表中，Pgs在TextVQA上最高比多数投票提升31.8分，但MatchedCtrl在所有基准上与之相当或更优，包括需要视觉理解的ViLP；所有Qwen类别均未显示显著优势。稳定性差距真实存在且依赖图像（最高达+0.48），但无法预测单个实例的胜出。结论为否定性诊断：扰动一致性至多是视觉依赖的部分诊断指标，在格式控制下无法作为可用筛选信号；此前报告的对比链式思维多数投票的增益被夸大。
原文摘要 · Abstract (English)
Test-time scaling lifts large language model reasoning by sampling many candidate solutions and selecting among them, yet the same recipe transfers poorly to vision-language models (VLMs): recent work shows that simple majority voting beats selection methods built on the model's own self-verification, apparently because at the selection layer an image-grounded answer and a confident guess from the language prior look the same. A natural fix is to make the selection signal one that cannot be computed without the image. We study Perturbation Grounded Selection (Pgs), a label-free, training-free rule that scores each candidate by whether the model re-derives it under label-preserving perturbations of the input (cropping, background masking, mild photometric or geometric jitter); Pgs recovers majority voting when the perturbation set is empty. The decisive question is not whether Pgs beats chain-of-thought only majority voting, but whether the perturbation term adds anything once decoding format and budget are controlled. We therefore introduce a format-matched control (MatchedCtrl): the same short, no-CoT draws spent on the original image. Across TextVQA, MATH-Vision, MMMU, and ViLP, with a Qwen headline (three-seed means) and LLaVA-OneVision coverage in matched-budget selector tables, Pgs appears to beat plain majority voting by up to +31.8 points on TextVQA (Qwen), but MatchedCtrl tracks or exceeds Pgs within noise on every benchmark, including the vision-required ViLP; no Qwen category shows a significant gain over this control. The stability gap is real and image-dependent (up to +0.48), yet does not predict per-instance wins. The result is negative and diagnostic: perturbation consistency is at best a partial diagnostic of visual dependence and, on its own, not a usable selection signal once format is controlled; gains reported against CoT-only majority voting overstate such methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。