arXiv:2606.13156cs.CVcs.AI2026-06被引 1

让视觉模型反复修正定位框,看似有效,实则自欺欺人。

Iterative Visual Thinking and the Self-Correction Mirage in VLM Grounding

论文配图:Iterative Visual Thinking and the Self-Correction Mirage in VLM Grounding
图 1 · 摘自论文原文
  • 用教师模型生成纠错数据,无需人工标注,训练迭代修正策略。
  • 报告的2.4%提升依赖隐藏的最优步骤选择,实际部署时无效。
  • 模型无法识别自己哪一步正确,适合研究评估机制的人看。

让视觉语言模型(VLM)在测试时更长时间思考,已推动近期进展。将此思想应用于空间定位,可通过视觉自校正实现:模型预测边界框,观察其在图像上的渲染结果,并多步优化。我们构建了忠实的实现——迭代视觉思维(IVT),采用两阶段方法:首先通过监督预热,利用基础模型自身预测作为真实误差,由教师VLM生成纠正推理轨迹(无需人工标注);随后使用GRPO与简单交并比(IoU)奖励进行训练。按常规报告方式衡量,该模型性能超越单次预测基线2.4个百分点([email protected])。但我们揭示这一提升是测量幻象:报告中对每样本保留最接近真实框的步骤,这相当于隐含使用了需要预测答案的预言者。若改用可部署、无标签的停止规则重新评估,性能提升消失,最佳策略并非迭代,而是直接停在第0步——此时表现与基线相当,且优于所有可上线的规则。根本原因在于验证失败:模型虽能在轨迹中生成更优框,却无法识别。自验证置信度与正确性相关性极弱(r≈0.22),反事实叠加实验表明,循环行为响应的是渲染框的存在而非其正确性。我们总结经验,提出诚实轨迹评估协议:固定无标签策略下的准确率,加上明确的‘预言者-可上线’差距。

原文摘要 · Abstract (English)

Letting a vision-language model (VLM) think longer at test time has driven much recent progress. A natural way to bring this to spatial grounding is visual self-correction: the model predicts a bounding box, sees it rendered on the image, and refines it over several steps. We build a faithful instance of this idea, Iterative Visual Thinking (IVT), with a two-phase recipe: a supervised warm-up in which the base model's own predictions serve as realistic errors that a teacher VLM turns into corrective reasoning traces (yielding training data without human annotation), followed by GRPO with a simple IoU reward. Measured the way such systems are usually reported, it works: the trained model surpasses the single-shot base by +2.4pp [email protected]. We show this gain is a measurement mirage. The reported number silently keeps, per sample, the trajectory step closest to the ground-truth box: an oracle that needs the very answer it predicts. Re-scored under deployable, label-free stopping rules the improvement vanishes, and the best policy is not to iterate at all: stopping at step 0 matches the base and beats every shippable rule. The cause is a verification failure, since the model can generate a better box somewhere in its trajectory but cannot identify it. Self-verification confidence correlates only weakly with correctness (r about 0.22), and a counterfactual overlay shows the loop reacts to the presence of a rendered box rather than its correctness. We distill the lesson into an honest-trajectory evaluation protocol: accuracy under fixed label-free policies plus an explicit oracle-shippable gap.

视觉定位自校正评估陷阱

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。