用代码转换图表,让大模型精准理解并推理复杂图表信息。
ChartReasoner: Code-Driven Modality Bridging for Long-Chain Reasoning in Chart Question Answering
- 将图表转为结构化ECharts代码,保留布局与数据语义
- 自动生成推理轨迹,在4个公开数据集上表现优异
- 适合需要高精度图表理解的科研、金融分析场景
近期大型语言模型在长链推理方面展现出强大能力,但如何将其拓展至视觉推理任务仍面临挑战。现有方法通过图像转文本的方式处理多模态推理,常丢失图表中关键的结构与语义信息,尤其在需大量视觉细节的任务如图表问答中表现不佳。为此,我们提出ChartReasoner,一种基于代码驱动的两阶段框架,实现对图表的精确、可解释推理。首先训练一个高保真模型,将多种图表图像转化为结构化ECharts代码,尽可能无损地保留布局与数据语义。随后设计通用的图表推理数据合成流水线,利用预训练转换模型自动且可扩展地生成推理路径,并通过代码验证器过滤低质量样本。最后,在合成数据集上结合监督微调与强化学习训练最终多模态模型。实验在四个公开基准上验证了该方法的有效性:能最大程度保留图表原始细节,参数更少却性能媲美顶尖开源模型,且在跨领域设置下接近GPT-4o等专有系统的表现。
原文摘要 · Abstract (English)
Recently, large language models have shown remarkable reasoning capabilities through long-chain reasoning before responding. However, how to extend this capability to visual reasoning tasks remains an open challenge. Existing multimodal reasoning approaches transfer such visual reasoning task into textual reasoning task via several image-to-text conversions, which often lose critical structural and semantic information embedded in visualizations, especially for tasks like chart question answering that require a large amount of visual details. To bridge this gap, we propose ChartReasoner, a code-driven novel two-stage framework designed to enable precise, interpretable reasoning over charts. We first train a high-fidelity model to convert diverse chart images into structured ECharts codes, preserving both layout and data semantics as lossless as possible. Then, we design a general chart reasoning data synthesis pipeline, which leverages this pretrained transport model to automatically and scalably generate chart reasoning trajectories and utilizes a code validator to filter out low-quality samples. Finally, we train the final multimodal model using a combination of supervised fine-tuning and reinforcement learning on our synthesized chart reasoning dataset and experimental results on four public benchmarks clearly demonstrate the effectiveness of our proposed ChartReasoner. It can preserve the original details of the charts as much as possible and perform comparably with state-of-the-art open-source models while using fewer parameters, approaching the performance of proprietary systems like GPT-4o in out-of-domain settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。