arXiv:2608.01930cs.CVcs.AI2026-08

发现视觉语言模型会依赖旧文本推理,提出方法阻止其重复使用过时信息。

Recompute or Reuse? Diagnosing and Mitigating Textual Shortcuts in VLM Self-Reflection

论文配图:Recompute or Reuse? Diagnosing and Mitigating Textual Shortcuts in VLM Self-Reflection
图 1 · 摘自论文原文
  • 识别出旧推理中的关键文本内容会形成行为捷径
  • 移除这些内容可使模型更依赖当前图像,更新率提升至53.61%
  • 适合关注模型可信推理与自我修正能力的研究者

视觉语言模型(VLMs)在视觉证据变化时应能修正其推理。现有研究常归因于视觉注意力不足或上下文惯性,却未明确模型复用的是什么而非重新计算。本文揭示,先前思维链(CoT)中承载证据的推理内容会形成文本捷径,与视觉重计算竞争。在16个VLM上,匹配反事实分析显示,此类内容是前序推理影响最稳定的载体。移除该内容比移除等长非证据上下文或最终答案段落更能改变答案偏好;随着更多过时证据被移除,先前推理的影响逐步减弱。重新排列该内容也削弱捷径强度,表明其结构影响捷径效力。此外,捷径在答案修正后仍具残留影响:当当前图像支持减弱时,模型倾向回退到先前答案;重复的先前答案和复用的前提主要出现在捷径活跃时。为抑制此影响,提出无需训练的Fresh-State Attention Firewall(FSAF),隔离新鲜计算与旧推理。在5个VLM上,FSAF将视觉更新率从35.28%提升至53.61%,先前答案占比从39.22%降至3.67%。可靠自反思不仅需重新观察,更需保护新视觉计算免受陈旧文本复用干扰。

原文摘要 · Abstract (English)

Vision-language models (VLMs) are expected to revise their reasoning when visual evidence changes. Failures to do so are often attributed to insufficient visual attention or contextual inertia, leaving unclear what models reuse instead of recomputing from the current image. We show that evidence-bearing reasoning in a prior chain of thought (CoT) can form a textual shortcut that competes behaviorally with visual recomputation. Across 16 VLMs, a matched counterfactual analysis identifies evidence-bearing content as the most robust carrier of prior-CoT influence. Removing this evidence-bearing content shifts answer preference more than removing length-matched non-evidence context or the final-answer span, with prior control weakening progressively as more stale evidence is removed. Reordering this evidence also weakens prior control, showing that its organization modulates shortcut strength. Beyond the immediate answer, the shortcut can retain residual influence after answer correction: weakening current-image support shifts preference back toward the prior answer, while repeated prior answers and reused premises arise mainly when the shortcut remains active. To limit this influence, we introduce Fresh-State Attention Firewall (FSAF), a training-free intervention that isolates fresh computation from the prior CoT. Across five VLMs, FSAF raises visual update rate from 35.28% to 53.61% and reduces prior-answer rate from 39.22% to 3.67%. Reliable VLM self-reflection therefore requires more than looking again: fresh visual recomputation must be protected from stale textual reuse.

视觉语言模型推理修正文本捷径自反思

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。