用工具协同推理,让AI读懂数学图像题
VisTIRA: Closing the Image-Text Modality Gap in Visual Math Reasoning via Structured Tool Integration
- 设计可分解图像数学题的工具协同推理框架
- 小模型靠OCR识别能显著提升图像解题准确率
- 大模型需结构化推理,小模型依赖文本对齐
视觉语言模型在以图像形式呈现的数学问题上表现远逊于纯文本。我们实证发现这是模态鸿沟:相同题目以文本形式解答准确率更高,源于公式密集、版式复杂和符号-图表混合上下文的多重读取失败。为此,提出VisTIRA(视觉与工具集成推理代理),通过迭代分解图像数学题为自然语言推理链和可执行Python步骤实现结构化求解。构建基于LaTeX的转换管道,将思维链数学语料(如NuminaMath)生成具挑战性的图像版本;并利用真实作业图像数据集SnapAsk生成大量合成工具使用轨迹,用于微调视觉语言模型。实验表明,工具集成监督能有效提升图像推理能力,且光学字符识别(OCR)对小模型有显著增益,但大模型下效果减弱。结果揭示模态鸿沟程度随模型规模增大而减小,结构化推理与基于OCR的对齐是互补策略。
原文摘要 · Abstract (English)
Vision-language models (VLMs) lag behind text-only language models on mathematical reasoning when the same problems are presented as images rather than text. We empirically characterize this as a modality gap: the same question in text form yields markedly higher accuracy than its visually typeset counterpart, due to compounded failures in reading dense formulas, layout, and mixed symbolic-diagrammatic context. First, we introduce VisTIRA (Vision and Tool-Integrated Reasoning Agent), a tool-integrated reasoning framework that enables structured problem solving by iteratively decomposing a given math problem (as an image) into natural language rationales and executable Python steps to determine the final answer. Second, we build a framework to measure and improve visual math reasoning: a LaTeX-based pipeline that converts chain-of-thought math corpora (e.g., NuminaMath) into challenging image counterparts, and a large set of synthetic tool-use trajectories derived from a real-world, homework-style image dataset (called SnapAsk) for fine-tuning VLMs. Our experiments show that tool-integrated supervision improves image-based reasoning, and OCR grounding can further narrow the gap for smaller models, although its benefit diminishes at scale. These findings highlight that modality gap severity inversely correlates with model size, and that structured reasoning and OCR-based grounding are complementary strategies for advancing visual mathematical reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。