用工具引导让视觉语言模型看穿视觉错觉
Seeing the Evidence, Missing the Answer: Tool-Guided Vision-Language Models on Visual Illusions
- 给模型配通用图像工具+路由提示,动态调用工具分析错觉
- 在未见过的错觉变体上仍保持稳定表现,泛化能力强
- 适合研究模型对错觉的鲁棒性或可解释性的人看
视觉语言模型在面对经典视觉错觉时存在系统性偏差:无论图像是否经过反事实修改,都倾向于将错觉判断为‘真实’。我们提出一种无需训练的工具引导推理框架,用于DataCV 2026挑战赛(任务一和任务二)。该框架让现成的视觉语言模型访问一组通用图像处理工具:线条绘制、区域裁剪、并排比较、通道隔离,配合基于错觉类型路由的提示系统,决定每类感知问题应调用哪些工具。关键在于,每次工具调用生成的新图像资源被永久存入持久注册表,模型可在推理链中随时引用和组合此前的标注视图。不依赖特定错觉的模块设计,实现了强跨结构泛化:从验证集到包含结构陌生错觉变体(如马赫带从竖向转为横向堆叠)的测试集,性能保持一致。我们还报告三个值得进一步研究的发现:(i) 强正向检测偏差,可能源于训练数据中的错觉不平衡;(ii) 像素级空间推理与自生成标注上的逻辑推理之间存在显著脱节;(iii) 对图像压缩伪影高度敏感,加剧了误报现象。
原文摘要 · Abstract (English)
Vision-language models (VLMs) exhibit a systematic bias when confronted with classic optical illusions: they overwhelmingly predict the illusion as "real" regardless of whether the image has been counterfactually modified. We present a tool-guided inference framework for the DataCV 2026 Challenge (Tasks I and II) that addresses this failure mode without any model training. An off-the-shelf vision-language model is given access to a small set of generic image manipulation tools: line drawing, region cropping, side-by-side comparison, and channel isolation, together with an illusion-type-routing system prompt that prescribes which tools to invoke for each perceptual question category. Critically, every tool call produces a new, immutable image resource appended to a persistent registry, so the model can reference and compose any prior annotated view throughout its reasoning chain. Rather than hard-coding illusion-specific modules, this generic-tool-plus-routing design yields strong cross-structural generalization: performance remained consistent from the validation set to a test set containing structurally unfamiliar illusion variants (e.g., Mach Bands rotated from vertical to horizontal stacking). We further report three empirical observations that we believe warrant additional investigation: (i) a strong positive-detection bias likely rooted in imbalanced illusion training data, (ii) a striking dissociation between pixel-accurate spatial reasoning and logical inference over self-generated annotations, and (iii) pronounced sensitivity to image compression artifacts that compounds false positives.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。