让多模态模型在推理时能反复查看图像，提升理解准确率。
TVI-CoT: Text-Visual Interleaved Chain-of-Thought Reasoning for Multimodal Understanding
- 用可学习的<THINK><LOOK><ANSWER> tokens实现文本与视觉的交替推理
- 在8个基准上显著提升性能，最高增益6.1%
- 适合需要精细视觉分析的多模态问答任务
思维链（CoT）推理在大语言模型中表现优异，但应用于多模态大模型（MLLM）时存在根本缺陷：推理过程完全依赖文本，无法访问图像特征。初始视觉编码后，图像信息即被丢弃，导致模型仅基于初始描述进行推理，形成‘视觉盲区’，限制了细粒度视觉提取、错误验证和自适应注意力。我们提出文本-视觉交织思维链（TVI-CoT），通过可学习控制标记<THINK>、<LOOK>和<ANSWER>，实现文本推理与视觉特征访问的显式交替。这些标记使模型能根据动态推理状态，动态切换至相关图像区域进行视觉定位。在八个基准上的实验表明，该方法在基于MLLM的CoT方法中达到顶尖水平，相比基线有显著提升：MMMU上+6.1%，MathVerse上+3.8%，MathVista上+3.4%，ScienceQA上+3.4%。代码已开源。
原文摘要 · Abstract (English)
Chain-of-thought (CoT) reasoning has proven effective for enhancing problem-solving in large language models. However, when applied to multimodal LLMs (MLLMs), existing CoT approaches suffer from a fundamental limitation: they perform reasoning entirely in text without accessing visual features during the reasoning process. After initial visual encoding, image information becomes inaccessible, forcing models to reason based solely on whatever was captured in the initial description, which forms a `vision-blind reasoning' paradigm that limits fine-grained visual extraction, error verification, and adaptive attention. We propose Text-Visual Interleaved Chain-of-Thought (TVI-CoT), a framework that enables explicit interleaving of textual reasoning and visual feature access through learnable control tokens <THINK>, <LOOK> and <ANSWER>. These tokens allow dynamic switching between reasoning and visual grounding, attending to relevant image regions conditioned on the evolving reasoning state. Experiments on eight benchmarks demonstrate state-of-the-art results among MLLM-based CoT methods and notable performance boost compared to the baseline: +6.1% on MMMU, +3.8% on MathVerse, +3.4% on MathVista, and +3.4% on ScienceQA. Code is available at https://github.com/hulianyuyy/TVI-CoT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。