评测大模型对图表各图层的理解与编辑能力,发现前后遮挡关系是难点。
Charts Are Beyond Pixels: Probing for Layer-Wise Chart Understanding and Editing

- 基于图层属性、绑定关系和可见性顺序构建新评测基准
- 大模型在图层归属上准确率达93%,但遮挡顺序仅61%正确
- 图像编辑器在遮挡约束下性能极低,提示需强化可见性建模
图表是具有功能角色、语义绑定和可见性关系的结构化视觉组合。现有评测主要关注输出结果的正确性,未直接评估图层级行为。我们提出LayerWiseBench,围绕图层归属、图层绑定和可见性顺序三个核心概念构建图表理解与编辑评测。所有图表由可执行的绘图程序生成,配套提供逐层对齐的RGBA资产及功能角色、语义绑定、可见性关系标签。基于此,设计可控的理解问题、编辑目标、参考图像与评估区域。数据集包含2,800个源图表,覆盖14种图表类型,衍生出7,329个图层理解问题和53,791个指令引导的编辑变体。在评估的VLM中,Qwen3.5-27B在问答任务中表现最优,图层归属准确率为93.04%,图层绑定为97.46%,但可见性顺序仅为61.46%。四款图像编辑器总体mIoU在1.49%至4.93%之间,遮挡约束编辑的mIoU最低,为0.37%至2.00%。结果表明,重叠组件的前后关系是理解和编辑中的持续挑战,亟需更显式的组件身份与可见性建模。
原文摘要 · Abstract (English)
Charts are structured visual compositions whose elements have distinct functional roles, semantic correspondences, and visibility relations. This structural view motivates evaluating whether models can understand and manipulate charts at the layer level. Existing chart benchmarks, however, primarily assess the correctness or fidelity of final outputs and do not directly evaluate these layer-wise behaviors. We present LayerWiseBench, a benchmark organized around three core concepts, layer attribution, layer binding, and visibility ordering, that structure its chart-understanding and chart-editing evaluations. Generated from executable chart programs, LayerWiseBench pairs each rendered chart with spatially aligned per-layer RGBA assets and construction-derived labels for functional roles, semantic bindings, and visibility relations. From this layer-wise representation, we derive controlled understanding questions, editing targets, reference images, and evaluation regions. It contains 2,800 source charts across 14 chart paradigms, from which we derive 7,329 layer-wise understanding questions and 53,791 instruction-guided editing variants. Among the evaluated VLMs, Qwen3.5-27B, which achieves the highest QA macro-average, obtains 93.04% accuracy on layer attribution and 97.46% on layer binding, but only 61.46% on visibility ordering. Across the four evaluated image editors, overall mIoU ranges from 1.49% to 4.93%, and visibility-constrained edits have the lowest mIoU for every editor, ranging from 0.37% to 2.00%. Taken together, these results identify tasks involving front-to-back relations between overlapping components as a recurring challenge across understanding and editing, motivating more explicit modeling of component identity and visibility relations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。