让AI推理时混用文字和视觉草图,提升复杂多模态问题解决能力。
Learning Modal-Mixed Chain-of-Thought Reasoning with Latent Embeddings
- 用视觉潜空间编码图像草图,与文字交错进行多模态推理。
- 在11个任务上超越纯文本CoT方法,关键在于视觉细节与高层意图分离。
- 适合需要跨模态分析的AI系统开发者,如视觉问答、场景理解等。
我们研究如何将链式思维(CoT)从语言扩展到多模态,以更好处理视觉密集型问题。传统CoT仅依赖文本,难以表达视觉中间状态。本文提出模态混合的CoT,将文本标记与紧凑的视觉草图(以潜空间嵌入表示)交替使用。为在不破坏视觉语言模型(VLM)原有知识的前提下弥合模态差距,我们使用VLM自身作为编码器,并训练语言主干重建其自身的中间视觉嵌入,确保视觉潜空间语义对齐。进一步引入基于扩散的潜解码器,由特殊控制标记触发,条件于VLM的隐藏状态。由此,扩散头负责精细感知细节,而VLM专注高层意图,实现角色解耦并减轻优化压力。训练分两阶段:第一阶段在交织文本与潜变量的推理轨迹上进行监督微调,采用联合下一个词与潜变量重构目标;第二阶段通过强化学习,学习何时切换模态及如何构建长链条推理。在11个多样化多模态推理任务上的实验表明,该方法性能优于纯语言和现有CoT方法。代码将公开发布。
原文摘要 · Abstract (English)
We study how to extend chain-of-thought (CoT) beyond language to better handle multimodal reasoning. While CoT helps LLMs and VLMs articulate intermediate steps, its text-only form often fails on vision-intensive problems where key intermediate states are inherently visual. We introduce modal-mixed CoT, which interleaves textual tokens with compact visual sketches represented as latent embeddings. To bridge the modality gap without eroding the original knowledge and capability of the VLM, we use the VLM itself as an encoder and train the language backbone to reconstruct its own intermediate vision embeddings, to guarantee the semantic alignment of the visual latent space. We further attach a diffusion-based latent decoder, invoked by a special control token and conditioned on hidden states from the VLM. In this way, the diffusion head carries fine-grained perceptual details while the VLM specifies high-level intent, which cleanly disentangles roles and reduces the optimization pressure of the VLM. Training proceeds in two stages: supervised fine-tuning on traces that interleave text and latents with a joint next-token and latent-reconstruction objective, followed by reinforcement learning that teaches when to switch modalities and how to compose long reasoning chains. Extensive experiments across 11 diverse multimodal reasoning tasks, demonstrate that our method yields better performance than language-only and other CoT methods. Our code will be publicly released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。