用代码生成图像辅助数学推理,让模型学会‘画图思考’。
CodePlot-CoT: Mathematical Visual Reasoning by Thinking with Code-Driven Images
- 通过可执行代码生成图像作为视觉思维,实现图文交替推理。
- 在17.8万样本的Math-VR数据集上,性能比基线提升21%。
- 适合需要画图辅助的数学问题求解,推动多模态推理发展。
大型语言模型和视觉语言模型在数学推理方面取得显著进展,但仍面临需视觉辅助的问题,如作辅助线或绘制函数图像。现有模型多限于纯文本推理链,而能生成交错文本与图像的统一多模态模型缺乏精度与可控性。为此,我们提出CodePlot-CoT——一种以代码驱动的“画图思考”链式推理范式。该方法利用视觉语言模型生成文本推理与可执行绘图代码,代码渲染为图像作为“视觉思维”以解决数学问题。首先,构建首个大规模双语数学视觉推理数据集Math-VR,包含178,000个样本;其次,开发先进的图像转代码转换器,专门解析复杂数学图形为代码;最后,基于此训练CodePlot-CoT模型。实验表明,该模型在新基准上相比基线最高提升21%,充分验证了代码驱动推理的有效性。本工作开辟了多模态数学推理新方向,并向社区提供首个大规模数据集、完整基准与强效方法。相关数据、代码及预训练模型已公开:https://github.com/HKU-MMLab/Math-VR-CodePlot-CoT。
原文摘要 · Abstract (English)
Recent advances in Large Language Models (LLMs) and Vision Language Models (VLMs) have shown significant progress in mathematical reasoning, yet they still face a critical bottleneck with problems requiring visual assistance, such as drawing auxiliary lines or plotting functions to solve the problems. Most LLMs and VLMs are constrained to text-only reasoning chains, while multimodal unified models that can generate interleaved text and images lack the necessary precision and controllability for such tasks. To address this, we propose CodePlot-CoT, a code-driven Chain-of-Thought paradigm for "thinking with images" in mathematics. Our approach leverages the VLM to generate text reasoning as well as executable plotting code, which is then rendered into images as "visual thought", to solve mathematical problems. To achieve this, we first construct Math-VR, the first large-scale, bilingual dataset and benchmark for Mathematics problems with Visual Reasoning, comprising 178K samples. Second, to create high-quality training data, we develop a state-of-the-art image-to-code converter specialized for parsing complex mathematical figures into codes. Finally, using these training data, we train the CodePlot-CoT model for solving mathematical problems. Experimental results show that our model achieves up to 21% increase over base model on our new benchmark, fully validating the efficacy of our proposed code-driven reasoning paradigm. Our work opens a new direction for multimodal mathematical reasoning and provides the community with the first large-scale dataset, comprehensive benchmark, and strong approach for such problems. To facilitate future research, we make our datasets, code, and pretrained models publicly available at https://github.com/HKU-MMLab/Math-VR-CodePlot-CoT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。