arXiv:2605.16848cs.CVcs.AI2026-05

让视觉模型通过识别模式,高效规划复杂任务。

Thinking with Patterns: Breaking the Perceptual Bottleneck in Visual Planning via Pattern Induction

论文配图:Thinking with Patterns: Breaking the Perceptual Bottleneck in Visual Planning via Pattern Induction
图 1 · 摘自论文原文
  • 用模式归纳法让模型自动发现可复用的视觉模式
  • 在多个环境中实现高准确率且计算开销更低的规划
  • 适合需要高效视觉决策的智能体研究者

从原始视觉输入进行规划仍是当前视觉语言模型(VLMs)的重大挑战,尤其当输入复杂度超出其单步感知能力时。受‘以图思考’(TWI)启发,我们提出将感知过程分解为逐步获取局部视觉证据的迭代方式。然而,尽管现有VLMs具备较强的通用TWI能力,其在规划任务中的感知瓶颈依然存在。为此,我们将TWI视为构建并更新内部世界模型的工具,发现无需训练的策略可使VLM解决远超其初始能力的任务。但过多的TWI操作会显著增加计算开销。为此,我们提出模式推理(Pattern Inference),使VLM能主动识别新任务中的已知视觉模式,并直接推断局部世界模型结构。为获得这些模式,我们设计在线归纳学习策略——模式归纳(Pattern Induction),将视觉模式视为可组合、可重用的专家,自主从经验中发现并优化。在FrozenLake、Crafter和CubeBench环境中的实验表明,该方法在准确率与效率间取得良好平衡。

原文摘要 · Abstract (English)

Planning from raw visual input remains a significant challenge for current Vision-Language Models (VLMs), when the complexity of input is beyond their one-step perception capability. Motivated by recent advances in Thinking with Images (TWI), a reasonable solution is to decompose the perception process into simpler steps by iteratively acquiring and incorporating local visual evidence. However, even though current VLMs are well-trained in general TWI ability, their perceptual bottleneck in the planning domain remains. To tackle this challenge, we formulate TWI as a tool to gradually build and reflect an accurate internal world model. We find that the resulting training-free planning strategy enables VLMs to solve tasks that are far beyond their initial capabilities, at the cost that too many TWI operations would significantly increase the computational overhead. To further improve efficiency, we propose Pattern Inference, a novel TWI strategy enabling VLMs to actively recognize known visual patterns in the new tasks and directly infer local world model structures. To obtain these patterns, we propose Pattern Induction, an online inductive learning strategy treating visual patterns as composite and reusable experts, which are autonomously discovered and optimized from experience. Experimental evaluations in FrozenLake, Crafter and CubeBench domains show that our approaches achieve a desirable balance between accuracy and efficiency.

视觉规划模式识别高效推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。