让大模型在潜在空间中用视觉思维推理，提升细节理解能力。
LanteRn: Latent Visual Structured Reasoning
- 用连续视觉思维嵌入实现语言与潜伏图像的交替推理。
- 在三个视觉任务上显著提升细粒度理解与定位准确率。
- 适合需要精准空间分析的多模态应用开发者使用。
尽管语言推理模型在诸多任务中表现优异，当前大型多模态模型（LMMs）在视觉推理方面仍面临挑战。多数LMMs仅将感知内容转为文本描述，限制了对精细空间与视觉信息的理解。现有方法虽尝试通过调用工具或生成中间图像实现图像内思考，但依赖外部模块或直接在像素空间推理，导致计算开销大。本文提出LanteRn框架，使LMMs能在推理过程中交替使用语言与紧凑的潜伏视觉表示，实现直接在潜伏空间中的视觉推理。该框架通过两阶段训练：监督微调以将视觉特征锚定于潜伏状态，再通过强化学习对齐潜伏推理与任务目标。在三个以感知为核心的基准测试（VisCoT、V* 和 Blink）上评估，结果均显示视觉定位与细粒度推理能力持续提升。这表明内部潜伏表示是更高效多模态推理的可行方向。
原文摘要 · Abstract (English)
While language reasoning models excel in many tasks, visual reasoning remains challenging for current large multimodal models (LMMs). As a result, most LMMs default to verbalizing perceptual content into text, a strong limitation for tasks requiring fine-grained spatial and visual understanding. While recent approaches take steps toward thinking with images by invoking tools or generating intermediate images, they either rely on external modules, or incur unnecessary computation by reasoning directly in pixel space. In this paper, we introduce LanteRn, a framework that enables LMMs to interleave language with compact latent visual representations, allowing visual reasoning to occur directly in latent space. LanteRn augments a vision-language transformer with the ability to generate and attend to continuous visual thought embeddings during inference. We train the model in two stages: supervised fine-tuning to ground visual features in latent states, followed by reinforcement learning to align latent reasoning with task-level utility. We evaluate LanteRn on three perception-centric benchmarks (VisCoT, V*, and Blink), observing consistent improvements in visual grounding and fine-grained reasoning. These results suggest that internal latent representations provide a promising direction for more efficient multimodal reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。