让模型用隐空间视觉符号进行视觉推理，不生成图像也能提升多模态理解能力
Machine Mental Imagery: Empower Multimodal Reasoning with Latent Visual Tokens
- 引入隐式视觉标记，在文本解码中穿插视觉符号推进多模态推理
- 在多个基准上表现优于依赖图像生成的模型，推理能力更强
- 适合需要深度视觉想象的任务，如复杂视觉问答与跨模态推理
视觉语言模型（VLMs）在多模态理解上表现出色，但其仅限文本解码的机制迫使模型将视觉推理转化为语言描述，限制了需视觉想象任务的表现。现有方法尝试训练模型生成显式图像，但图像生成预训练常损害推理能力。受人类通过心理意象（内部构建和操作视觉线索）进行推理的启发，我们探索无需生成像素级图像的多模态推理路径。为此，提出名为Mirage的机器心理意象框架，通过在普通文本外添加隐式视觉标记来增强VLM解码。当模型决定‘视觉思考’时，将其隐藏状态转为下一轮的隐式视觉标记，从而延续多模态轨迹。首先通过从真实图像嵌入中蒸馏监督隐式标记，随后切换至纯文本监督以使隐式轨迹紧密对齐任务目标，最后通过强化学习阶段进一步提升多模态推理能力。实验表明，Mirage在多个基准上实现了更强的多模态推理能力，且无需生成显式图像。
原文摘要 · Abstract (English)
Vision-language models (VLMs) excel at multimodal understanding, yet their text-only decoding forces them to verbalize visual reasoning, limiting performance on tasks that demand visual imagination. Recent attempts train VLMs to render explicit images, but the heavy image-generation pre-training often hinders the reasoning ability. Inspired by the way humans reason with mental imagery-the internal construction and manipulation of visual cues-we investigate whether VLMs can reason through interleaved multimodal trajectories without producing explicit images. To this end, we present a Machine Mental Imagery framework, dubbed as Mirage, which augments VLM decoding with latent visual tokens alongside ordinary text. Concretely, whenever the model chooses to ``think visually'', it recasts its hidden states as next tokens, thereby continuing a multimodal trajectory without generating pixel-level images. Begin by supervising the latent tokens through distillation from ground-truth image embeddings, we then switch to text-only supervision to make the latent trajectory align tightly with the task objective. A subsequent reinforcement learning stage further enhances the multimodal reasoning capability. Experiments on diverse benchmarks demonstrate that Mirage unlocks stronger multimodal reasoning without explicit image generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。