评测大模型多轮图表编辑能力，发现其易因上下文丢失而出错。
ChartEditBench: Evaluating Grounded Multi-Turn Chart Editing in Multimodal Language Models
- 构建5000条可控难度的多轮图表修改链，支持持续交互评估。
- 多轮编辑中错误累积严重，数据类变换执行失败率高。
- 适合研究多模态编程、交互式数据分析的学者使用。
尽管多模态大语言模型在单轮图表生成上表现良好，但其在真实世界探索性数据分析中的能力仍待深入研究。实践中，用户通过多轮交互逐步优化可视化，需保持共同语境、追踪历史修改并适应动态偏好。本文提出ChartEditBench，一个基于代码的增量式、视觉锚定图表编辑基准，包含5000条难度可控的修改链及严格人工验证子集。不同于以往一次性评估，该基准聚焦持续、上下文感知的编辑能力。我们进一步设计了稳健的评估框架，通过执行结果验证、像素级视觉相似度和逻辑代码校验，克服了LLM作为评判者带来的局限性。实验表明，当前先进MLLM在多轮场景下性能显著下降，主要源于错误累积与共享上下文失效，虽在风格编辑上表现良好，但在数据相关变换中频繁出现执行失败。ChartEditBench为具身化、意图感知的多模态编程提供了挑战性测试平台。
原文摘要 · Abstract (English)
While Multimodal Large Language Models (MLLMs) perform strongly on single-turn chart generation, their ability to support real-world exploratory data analysis remains underexplored. In practice, users iteratively refine visualizations through multi-turn interactions that require maintaining common ground, tracking prior edits, and adapting to evolving preferences. We introduce ChartEditBench, a benchmark for incremental, visually grounded chart editing via code, comprising 5,000 difficulty-controlled modification chains and a rigorously human-verified subset. Unlike prior one-shot benchmarks, ChartEditBench evaluates sustained, context-aware editing. We further propose a robust evaluation framework that mitigates limitations of LLM-as-a-Judge metrics by integrating execution-based fidelity checks, pixel-level visual similarity, and logical code verification. Experiments with state-of-the-art MLLMs reveal substantial degradation in multi-turn settings due to error accumulation and breakdowns in shared context, with strong performance on stylistic edits but frequent execution failures on data-centric transformations. ChartEditBench, establishes a challenging testbed for grounded, intent-aware multimodal programming.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。