让AI看懂图表，通过工具增强推理能力
CharTool: Tool-Integrated Visual Reasoning for Chart Understanding
- 用合成与真实图表构建训练数据，提升模型理解力
- 引入图像裁剪和代码计算工具，精准处理图表数字
- 在多个评测中超越大模型，适合需要图表分析的场景
图表广泛用于科学与金融文献中呈现结构化数据。然而，由于缺乏高质量训练数据，以及对细粒度视觉定位和精确数值计算的需求，多模态大语言模型（MLLMs）在图表推理上仍面临挑战。为此，我们首先提出DuoChart，一种可扩展的双源数据管道，将合成图表与真实图表结合，构建多样且高质量的训练数据。随后，我们引入CharTool，为MLLMs配备外部工具，包括图像裁剪以实现局部视觉感知，以及基于代码的计算以保证数值推理准确性。通过在DuoChart上进行代理式强化学习，CharTool学习到基于图表内容的工具集成推理能力。在六个图表基准上的大量实验表明，该方法在不同规模模型下均显著优于强基线。特别地，CharTool-7B在CharXiv（Reasoning）上比基础模型提升+8.0%，在ChartQAPro上提升+9.78%，性能媲美更大或专有模型。此外，CharTool在跨领域视觉数学推理任务中也表现出良好泛化能力。
原文摘要 · Abstract (English)
Charts are ubiquitous in scientific and financial literature for presenting structured data. However, chart reasoning remains challenging for multimodal large language models (MLLMs) due to the lack of high-quality training data, as well as the need for fine-grained visual grounding and precise numerical computation. To address these challenges, we first propose DuoChart, a scalable dual-source data pipeline that combines synthesized charts with real-world charts to construct diverse, high-quality chart training data. We then introduce CharTool, which equips MLLMs with external tools, including image cropping for localized visual perception and code-based computation for accurate numerical reasoning. Through agentic reinforcement learning on DuoChart, CharTool learns tool-integrated reasoning grounded in chart content. Extensive experiments on six chart benchmarks show that our method consistently improves over strong MLLM baselines across model scales. Notably, CharTool-7B outperforms the base model by +8.0% on CharXiv (Reasoning) and +9.78% on ChartQAPro, while achieving competitive performance with substantially larger or proprietary models. Moreover, CharTool demonstrates positive generalization to out-of-domain visual math reasoning benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。