arXiv:2508.17608cs.LG2025-08被引 15

用真实论文图表训练模型,让生成代码更贴近原图样式。

ChartMaster: Advancing Chart-to-Code Generation with Real-World Charts and Chart Similarity Reinforcement Learning

  • 用arXiv真实图表构建大规模数据集,提升多样性。
  • 设计视觉相似性奖励,使生成图表与原图高度一致。
  • 适合需要高保真图表生成的开发者和研究者。

图表转代码任务要求多模态大模型将图表图像转换为可执行代码。该任务面临两大挑战:数据多样性不足,以及生成图表与原始图表的视觉一致性难以保证。现有数据集主要依赖合成种子数据生成代码,导致样本同质化,限制模型对真实图表风格的泛化能力。为此,我们提出ReChartPrompt,利用从arXiv论文中提取的真实人类设计图表作为提示,构建了包含24万张图表的ReChartPrompt-240K数据集,显著提升数据多样性和真实性。针对第二项挑战,尽管监督微调(SFT)能改善代码理解,但其仅优化下一个词预测,未直接监督视觉特征,导致生成图表常与原图不一致。为此,我们提出ChartSimRL,一种基于GRPO的强化学习算法,采用新型图表相似性奖励。该奖励由两部分构成:属性相似性(衡量布局、颜色等属性重叠),以及视觉相似性(使用卷积神经网络评估纹理等整体视觉特征)。与传统文本奖励不同,本奖励充分考虑图表生成的多模态特性,显著提升模型还原图表的能力。结合ReChartPrompt与ChartSimRL,我们构建了ChartMaster模型,在7B参数模型中达到当前最佳性能,并在多个基准上媲美GPT-4o。所有资源已公开于https://github.com/WentaoTan/ChartMaster。

原文摘要 · Abstract (English)

The chart-to-code generation task requires MLLMs to convert chart images into executable code. This task faces two main challenges: limited data diversity and the difficulty of maintaining visual consistency between generated charts and the original ones. Existing datasets mainly rely on synthetic seed data to prompt GPT models for code generation, resulting in homogeneous samples that limit model generalization to real-world chart styles. To address this, we propose ReChartPrompt, leveraging real-world, human-designed charts extracted from arXiv papers as prompts. By harnessing the rich content and diverse visual styles of arXiv charts, we construct ReChartPrompt-240K, a large-scale and highly diverse dataset that better reflects realistic chart variations. For the second challenge, although SFT improves code understanding by optimizing next-token prediction, it does not provide direct supervision on visual features. As a result, it often fails to guarantee that the generated charts visually match the original ones. To address this, we propose ChartSimRL, a GRPO-based reinforcement learning algorithm guided by a novel chart similarity reward. This reward consists of two components: attribute similarity, which measures the overlap of chart attributes like layout and color between the generated and original charts, and visual similarity, which evaluates overall visual features, including texture, using convolutional neural networks. Unlike traditional text-based rewards, our reward accounts for the multimodal nature of the chart-to-code generation task, significantly enhancing the model's ability to accurately reproduce charts. Integrating ReChartPrompt and ChartSimRL, we develop the ChartMaster model, achieving SOTA results among 7B-parameter models and rivaling GPT-4o on various chart-to-code benchmarks. All resources are available at https://github.com/WentaoTan/ChartMaster.

图表生成多模态强化学习代码生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。