自动生成复杂图表推理数据,提升模型理解能力
ChartM$^3$: A Multi-Stage Code-Driven Pipeline for Constructing Multi-Dimensional and Multi-Step Visual Reasoning Data in Chart Comprehension
- 用代码驱动多阶段流程生成图表与问题对
- 构建38K图表、142K问答对及2,871评估样本
- 适合训练和评估复杂图表理解的模型
复杂图表理解任务需要多模态大语言模型具备高级视觉识别与推理能力。然而,现有研究对真实场景中常见的复杂图表与计算密集型推理任务覆盖有限。本文提出一种自动化的多阶段代码驱动流水线,系统生成视觉推理数据集。该流水线结合检索增强生成(RAG)获取专业图表模板,并采用思维链(CoT)策略生成模拟真实数据分布的推理代码,驱动图表渲染与相关统计计算。通过模型评估,该流程提升了图表多样性与数据质量。基于此框架,我们构建了ChartM$^3$,一个包含38,000张图表、142,000个问答对的多维多步数据集,以及2,871个高质量评估样本,支持实际性能评估。监督微调(SFT)与强化学习(RL)实验表明,该数据集显著提升模型推理能力与跨域泛化性能,使小模型在复杂图表理解上达到大模型水平。
原文摘要 · Abstract (English)
Complex chart understanding tasks demand advanced visual recognition and reasoning capabilities from multimodal large language models (MLLMs). However, current research provides limited coverage of complex chart scenarios and computation-intensive reasoning tasks prevalent in real-world applications. This study proposes an automated multi-stage code-driven pipeline for systematically generating visual reasoning datasets to address these limitations. The pipeline integrates retrieval-augmented generation (RAG) to retrieve professional chart templates and employs chain-of-thought (CoT) strategies to generate reasoning codes that simulate real data distributions, thereby driving chart rendering and question-related statistical computations. Through model-based evaluation, the pipeline enhances chart diversity and data quality. Using this framework, we construct ChartM$^3$, a multi-dimensional and multi-step dataset containing 38K charts and 142K Q&A pairs for training, along with 2,871 high-quality evaluation samples for enabling practical performance assessment. Supervised fine-tuning (SFT) and reinforcement learning (RL) experiments demonstrate that our dataset significantly improves reasoning capabilities and cross-domain generalization performance, enabling smaller models to achieve performance comparable to larger-scale models in complex chart comprehension.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。