arXiv:2607.00606cs.CV2026-07

不训练、不生成,用检索图像实现视觉推理,效果逼近顶尖模型。

Retrieved Images as Visual Thought: Training-Free Multimodal In-Context Learning for the Open-vs-Closed Gap

论文配图:Retrieved Images as Visual Thought: Training-Free Multimodal In-Context Learning for the Open-vs-Closed Gap
图 1 · 摘自论文原文
  • 通过检索带标签的图像作为视觉思维单元,结合多模态检索与对话轮转机制。
  • 4样本下准确率98.5%,参数量仅为72B模型的约1/2.4,性能接近最优。
  • 适合需要高效、免训练视觉推理的场景,尤其适合开放世界任务。

近期研究将视觉融入推理过程,但依赖生成:使用外部工具、编写代码或生成新图像,分别带来协议开销、代码脆弱或训练成本。另一种无需生成的路径是检索已标注图像并基于其推理,却长期未受重视。本文提出ReVisIT,一种免训练框架,将每个检索到的图像-标签对视为一个视觉思维单元。该框架结合结构化类别定义、针对每查询的多模态示例检索,以及用户/助手交替注入这些示例后联合解码多属性输出,并能根据任务适配降级使用。在VL-ICL Bench Fast Open MiniImageNet上,使用Qwen3-VL-30B-A3B的ReVisIT达到4样本98.5%准确率,与72B LLaVA-OneVision SOTA(98.7%)统计无差异,而参数量仅为其约1/2.4;相同骨干无框架时表现仅随机水平。单用‘回合层’使GPT-4.1在自由形式概念归纳(Bongard-OpenWorld)上提升26.1分,完整系统在三个骨干上于新发布的无版权MAAC-Bench(27类,5属性)实现4-6分宏观增益,经配对置换检验显著。组件分析表明,检索+回合是通用杠杆,结构化定义则按需使用;83%的检索收益来自检索质量而非示例存在本身。MAAC-Bench附带基于评分标准的LLM验证协议,替代人工主观抽查。

原文摘要 · Abstract (English)

Recent work on Thinking with Images makes vision a dynamic part of reasoning, but does so through generation: the model invokes external tools, synthesizes code, or imagines new imagery, each at the cost of a tool protocol, brittle code, or an expensive training pipeline. A fourth route makes vision dynamic without generating anything, by retrieving labeled exemplar images and reasoning over them, yet it remains underexplored despite being train-free. We present ReVisIT, a train-free framework that realizes this retrieval-based route by treating each retrieved image-label pair as a unit of visual thought. ReVisIT combines structured class definitions, per-query multimodal retrieval of exemplars, and alternating user/assistant injection of those exemplars before joint multi-attribute decoding, and degrades gracefully to whichever components a task admits. On VL-ICL Bench Fast Open MiniImageNet, Qwen3-VL-30B-A3B with ReVisIT reaches 98.5% at 4-shot, statistically indistinguishable from the 72B LLaVA-OneVision SOTA (98.7%) on this near-saturated task at about 1/2.4 the parameters, while the same backbone without the scaffold sits at chance. The turns layer alone adds 26.1 points to GPT-4.1 on free-form concept induction (Bongard-OpenWorld), and the full stack yields a 4-6 point macro gain across three backbones on MAAC-Bench, a new license-clean 27-class, 5-attribute benchmark, significant by paired bootstrap on the curator-derived attributes. Component analysis shows that retrieval-plus-turns is the universal lever while structured definitions are need-adaptive, and that 83% of the retrieval gain comes from retrieval quality rather than from the presence of exemplars. MAAC-Bench is released with a rubric-grounded LLM verification protocol that replaces author spot-check on subjective attributes.

视觉推理检索增强免训练多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。