arXiv:2501.06598cs.AI2025-01ACL被引 89

用代码解析图表,让生成的代码可运行且细节更完整。

ChartCoder: Advancing Multimodal Large Language Model for Chart-to-Code Generation

  • 用代码大模型做语言主干,提升代码可执行性。
  • 自建16万条数据集,支持多样化图表转代码任务。
  • 分步生成法让模型更精准还原图表细节,适合开发者使用。

多模态大语言模型在图表理解任务中表现优异,但仅依赖文本描述会丢失图表中的密集信息。将图表解析为代码可实现无损表示,保留所有关键细节。现有开源多模态模型在图表转代码任务中仍面临两大挑战:生成代码可执行性差、细节恢复不足;缺乏大规模多样化的训练数据。为此,我们提出首个专用于图表转代码的多模态大模型 ChartCoder,采用代码大模型作为语言主干以增强代码可执行性。同时,构建首个大规模多样化数据集 Chart2Code-160k,提出分步生成方法 Snippet-of-Thought(SoT),将直接生成任务转化为逐步推理。实验表明,仅70亿参数的 ChartCoder 在多个图表转代码基准上超越现有开源模型,显著提升图表还原度与代码可执行性。代码已开源。

原文摘要 · Abstract (English)

Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities in chart understanding tasks. However, interpreting charts with textual descriptions often leads to information loss, as it fails to fully capture the dense information embedded in charts. In contrast, parsing charts into code provides lossless representations that can effectively contain all critical details. Although existing open-source MLLMs have achieved success in chart understanding tasks, they still face two major challenges when applied to chart-to-code tasks: (1) Low executability and poor restoration of chart details in the generated code and (2) Lack of large-scale and diverse training data. To address these challenges, we propose \textbf{ChartCoder}, the first dedicated chart-to-code MLLM, which leverages Code LLMs as the language backbone to enhance the executability of the generated code. Furthermore, we introduce \textbf{Chart2Code-160k}, the first large-scale and diverse dataset for chart-to-code generation, and propose the \textbf{Snippet-of-Thought (SoT)} method, which transforms direct chart-to-code generation data into step-by-step generation. Experiments demonstrate that ChartCoder, with only 7B parameters, surpasses existing open-source MLLMs on chart-to-code benchmarks, achieving superior chart restoration and code excitability. Our code is available at https://github.com/thunlp/ChartCoder.

图表生成代码生成多模态指令微调

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。