让AI在解数学题时灵活穿插图像细节，提升推理准确率。
MINT-CoT: Enabling Interleaved Visual Tokens in Mathematical Chain-of-Thought Reasoning
- 用动态选择的视觉标记，将图像碎片融入文字推理过程。
- 在三个数据集上平均提升超20%，最高达34.08%。
- 适合需要图文联合推理的数学教育与智能辅导系统。
链式思维（CoT）已广泛提升大语言模型的数学推理能力，但在多模态领域仍面临挑战。现有方法或对图像输入采用类文本推理，或尝试将视觉信号嵌入数学CoT，但存在三大局限：依赖粗粒度框形图像区域、视觉编码器对数学内容感知有限、需外部工具修改视觉信息。本文提出MINT-CoT，引入数学链式思维中的交错视觉标记。MINT-CoT通过一个交错标记，自适应地将任意形状的数学图形视觉区域融入文本推理步骤。为支持该能力，我们构建了含5.4万道数学题的MINT-CoT数据集，每道题在标记级别对齐推理步骤与视觉区域，并配套严格的数据生成流程。我们还设计三阶段训练策略：文本仅微调、交错式微调、交错式强化学习，最终获得MINT-CoT-7B模型。大量实验表明，该方法在数学多模态推理中有效，模型在MathVista上比基线提升34.08%，在GeoQA上提升28.78%，在MMStar上提升23.2%。
原文摘要 · Abstract (English)
Chain-of-Thought (CoT) has widely enhanced mathematical reasoning in Large Language Models (LLMs), but it still remains challenging for extending it to multimodal domains. Existing works either adopt a similar textual reasoning for image input, or seek to interleave visual signals into mathematical CoT. However, they face three key limitations for math problem-solving: reliance on coarse-grained box-shaped image regions, limited perception of vision encoders on math content, and dependence on external capabilities for visual modification. In this paper, we propose MINT-CoT, introducing Mathematical INterleaved Tokens for Chain-of-Thought visual reasoning. MINT-CoT adaptively interleaves relevant visual tokens into textual reasoning steps via an Interleave Token, which dynamically selects visual regions of any shapes within math figures. To empower this capability, we construct the MINT-CoT dataset, containing 54K mathematical problems aligning each reasoning step with visual regions at the token level, accompanied by a rigorous data generation pipeline. We further present a three-stage MINT-CoT training strategy, progressively combining text-only CoT SFT, interleaved CoT SFT, and interleaved CoT RL, which derives our MINT-CoT-7B model. Extensive experiments demonstrate the effectiveness of our method for effective visual interleaved reasoning in mathematical domains, where MINT-CoT-7B outperforms the baseline model by +34.08% on MathVista, +28.78% on GeoQA, and +23.2% on MMStar, respectively. Our code and data are available at https://github.com/xinyan-cxy/MINT-CoT
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。