用双偏好引导优化,让AI更准地把图表转成代码。
Boosting Chart-to-Code Generation in MLLM via Dual Preference-Guided Refinement
- 设计双模态奖励机制,通过反馈迭代提升生成质量。
- 在多个开源模型上实现接近专业系统的代码生成效果。
- 适合需要高精度图表转代码的开发者和研究者。
将图表图像转化为可执行绘图代码的任务——即图表转代码——要求多模态大模型具备细粒度视觉解析、精确代码生成和强跨模态推理能力。然而,该任务本质上是欠约束的:同一张图表可对应多种有效代码实现,评估需兼顾代码正确性与多维度视觉保真度。这使得标准监督微调难以学习到准确且泛化的映射关系。为此,我们提出一种双偏好引导精炼框架,结合反馈驱动的双模态奖励机制与迭代偏好学习。方法引入结构化变体生成策略和视觉奖励模型,高效构建高质量、面向特定维度的偏好对,使偏好收集更具可扩展性且监督更精准。这些偏好用于离线强化学习,优化模型以实现多维度保真度。实验表明,该框架显著提升通用开源多模态大模型的表现,使其生成的绘图代码质量媲美专门设计的图表导向模型,甚至超越部分专有系统。代码与数据集已公开于 https://github.com/Zhihan72/Chart2Code。
原文摘要 · Abstract (English)
Translating chart images into executable plotting scripts-referred to as the chart-to-code generation task-requires Multimodal Large Language Models (MLLMs) to perform fine-grained visual parsing, precise code synthesis, and robust cross-modal reasoning. However, this task is inherently under-constrained: multiple valid code implementations can produce the same visual chart, and evaluation must consider both code correctness and visual fidelity across diverse dimensions. This makes it difficult to learn accurate and generalizable mappings through standard supervised fine-tuning. To address these challenges, we propose a dual preference-guided refinement framework that combines a feedback-driven, dual-modality reward mechanism with iterative preference learning. Our approach introduces a structured variant generation strategy and a visual reward model to efficiently produce high-quality, aspect-aware preference pairs-making preference collection scalable and supervision more targeted. These preferences are used in an offline reinforcement learning setup to optimize the model toward multi-dimensional fidelity. Experimental results show that our framework significantly enhances the performance of general-purpose open-source MLLMs, enabling them to generate high-quality plotting code that rivals specialized chart-centric models and even some proprietary systems. The code and datasets are publicly available at https://github.com/Zhihan72/Chart2Code.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。