arXiv:2512.16584cs.CV2025-12被引 18

让大模型在推理时直接生成连续视觉嵌入,实现图文混合思维。

Sketch-in-Latents: Eliciting Unified Reasoning in MLLMs

  • 将视觉想象融入文本推理过程,动态生成连续的潜在草图嵌入
  • 在视觉核心任务上表现更优,且在多模态基准上泛化能力强
  • 适合需要灵活图文交互的智能系统研发者

尽管多模态大语言模型(MLLMs)在基于文本推理的视觉理解任务中表现优异,但在需要视觉想象的场景中仍显不足。与现有依赖预定义外部工具包或在思考过程中生成图像的方法不同,人类能够在无需预设工具的情况下,在大脑内部统一空间中构建灵活的图文想象与互动。受此启发,鉴于当前MLLMs已将视觉与文本信息编码在同一特征空间,我们提出将视觉标记无缝插入由文本标记承载的推理过程,理想情况下所有视觉想象均可由潜在特征编码。为此,我们提出Sketch-in-Latents(SkiLa),一种新型统一多模态推理范式,扩展了MLLMs的自回归能力,原生生成连续视觉嵌入,称为潜在草图标记(latent sketch tokens),作为视觉思维。在多步推理中,模型动态交替使用文本思考模式生成文本思维标记和视觉草图模式生成潜在草图标记。同时提出潜在视觉语义重建机制,确保这些潜在草图标记具有语义基础。大量实验表明,SkiLa在以视觉为中心的任务上取得卓越性能,并在多样化的通用多模态基准上展现出强泛化能力。代码将发布于 https://github.com/TungChintao/SkiLa。

原文摘要 · Abstract (English)

While Multimodal Large Language Models (MLLMs) excel at visual understanding tasks through text reasoning, they often fall short in scenarios requiring visual imagination. Unlike current works that take predefined external toolkits or generate images during thinking, however, humans can form flexible visual-text imagination and interactions during thinking without predefined toolkits, where one important reason is that humans construct the visual-text thinking process in a unified space inside the brain. Inspired by this capability, given that current MLLMs already encode visual and text information in the same feature space, we hold that visual tokens can be seamlessly inserted into the reasoning process carried by text tokens, where ideally, all visual imagination processes can be encoded by the latent features. To achieve this goal, we propose Sketch-in-Latents (SkiLa), a novel paradigm for unified multi-modal reasoning that expands the auto-regressive capabilities of MLLMs to natively generate continuous visual embeddings, termed latent sketch tokens, as visual thoughts. During multi-step reasoning, the model dynamically alternates between textual thinking mode for generating textual think tokens and visual sketching mode for generating latent sketch tokens. A latent visual semantics reconstruction mechanism is proposed to ensure these latent sketch tokens are semantically grounded. Extensive experiments demonstrate that SkiLa achieves superior performance on vision-centric tasks while exhibiting strong generalization to diverse general multi-modal benchmarks. Codes will be released at https://github.com/TungChintao/SkiLa.

多模态推理视觉想象潜在空间大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。