arXiv:2505.11935cs.CL2025-05ACL被引 31

评测10个大模型编辑图表的能力,发现当前最先进模型准确率仅59.96%。

ChartEdit: How Far Are MLLMs From Automating Chart Analysis? Evaluating MLLMs' Capability via Chart Editing

  • 构建包含1405条指令的图表编辑基准集,覆盖233张真实图表。
  • 顶级模型在代码级和图像级均表现有限,准确率最高仅59.96%。
  • 小模型更差,说明图文理解与精准修改仍面临重大挑战。

尽管多模态大语言模型(MLLMs)在生成图表渲染代码方面展现出潜力,但通过代码编辑图表更具挑战性,需融合图表理解与推理能力,且耗时费力。虽许多MLLM声称具备此类能力,但现有评估依赖有限案例研究,亟需全面评价框架。本文提出 extsc{ChartEdit},一个新型图表编辑基准,包含1405条多样化的编辑指令应用于233张真实世界图表,每张均经人工标注与验证。利用 extsc{ChartEdit},我们在代码与图表两个层面评估了10个主流MLLM的表现。结果表明,大规模模型可生成部分匹配参考图像的代码,但严格按指令生成准确编辑的能力仍受限。当前最优模型得分仅为59.96%,凸显精确修改的重大挑战。相比之下,小型模型(包括图表领域专用模型)在遵循指令和生成整体图表图像方面均表现不佳,表明该领域仍需深入发展。代码已开源:https://github.com/xxlllz/ChartEdit。

原文摘要 · Abstract (English)

Although multimodal large language models (MLLMs) show promise in generating chart rendering code, editing charts via code presents a greater challenge. This task demands MLLMs to integrate chart understanding and reasoning capacities, which are labor-intensive. While many MLLMs claim such editing capabilities, current evaluations rely on limited case studies, highlighting the urgent need for a comprehensive evaluation framework. In this work, we propose \textsc{ChartEdit}, a novel benchmark designed for chart editing tasks, featuring $1405$ diverse editing instructions applied to $233$ real-world charts, each manually annotated and validated for accuracy. Utilizing \textsc{ChartEdit}, we evaluate the performance of 10 mainstream MLLMs across two types of experiments at both the code and chart levels. The results suggest that large-scale models can generate code to produce images that partially match the reference images. However, their ability to generate accurate edits according to the instructions remains limited. The state-of-the-art (SOTA) model achieves a score of only $59.96$, highlighting significant challenges in precise modification. In contrast, small-scale models, including chart-domain models, struggle both with following editing instructions and generating overall chart images, underscoring the need for further development in this area. Code is available at https://github.com/xxlllz/ChartEdit.

图表分析多模态模型评测基准

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。