让大模型像人一样画画思考,提升复杂视觉推理能力。
Latent Sketchpad: Sketching Visual Thoughts to Elicit Multimodal Reasoning in MLLMs
- 将模型内部视觉表征用于生成性视觉思维,边想边画。
- 在MazePlanning数据集上表现优于或媲美原模型,支持跨模型泛化。
- 适合需要创意规划、交互式推理的AI应用开发者。
尽管多模态大语言模型(MLLMs)在视觉理解方面表现优异,但在需要视觉规划与想象的复杂场景中仍显不足。受人类通过草图进行视觉思维的启发,我们提出Latent Sketchpad框架,为MLLMs配备内部视觉草稿板。传统上,模型的内部视觉表征仅限于感知理解,我们将其重用于生成性视觉思考,且不损害推理能力。基于前沿的MLLMs,本方法将视觉生成直接融入其自回归推理过程,使模型可交替进行文本推理与视觉潜空间生成。这些潜空间表征引导内部思维,并可转换为草图图像以增强可解释性。为此,我们引入两个组件:上下文感知视觉头(autoregressively生成视觉表示),以及预训练草图解码器(将潜空间转为可读图像)。我们在新构建的数据集MazePlanning上评估该框架。实验表明,无论在何种前沿MLLMs上,Latent Sketchpad均达到或超越原模型的推理性能,并展现出跨模型泛化能力,包括Gemma3和Qwen2.5-VL。通过扩展模型的文本推理为视觉思考,该框架为更丰富的交互体验与广泛的应用开辟了新路径。
原文摘要 · Abstract (English)
While Multimodal Large Language Models (MLLMs) excel at visual understanding, they often struggle in complex scenarios that require visual planning and imagination. Inspired by how humans use sketching as a form of visual thinking to develop and communicate ideas, we introduce Latent Sketchpad, a framework that equips MLLMs with an internal visual scratchpad. The internal visual representations of MLLMs have traditionally been confined to perceptual understanding. We repurpose them to support generative visual thought without compromising reasoning ability. Building on frontier MLLMs, our approach integrates visual generation directly into their native autoregressive reasoning process. It allows the model to interleave textual reasoning with the generation of visual latents. These latents guide the internal thought process and can be translated into sketch images for interpretability. To realize this, we introduce two components: a Context-Aware Vision Head autoregressively produces visual representations, and a pretrained Sketch Decoder renders these into human-interpretable images. We evaluate the framework on our new dataset MazePlanning. Experiments across various MLLMs show that Latent Sketchpad delivers comparable or even superior reasoning performance to their backbone. It further generalizes across distinct frontier MLLMs, including Gemma3 and Qwen2.5-VL. By extending model's textual reasoning to visual thinking, our framework opens new opportunities for richer human-computer interaction and broader applications. More details and resources are available on our project page: https://latent-sketchpad.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。