让大模型同时理解图表的布局和数据，提升精准分析能力。
START: Spatial and Textual Learning for Chart Understanding
- 通过元素定位与图表转代码，同步学习图表空间结构和数据内容。
- 在多个基准上超越现有方法，小模型也显著提效。
- 适合需要精细图表理解的科研、报告分析场景。
图表理解对部署多模态大语言模型（MLLM）于真实场景（如科学论文和技术报告分析）至关重要。与自然图像不同，图表兼具结构化视觉布局（空间属性）与底层数据表示（文本属性），需同时掌握才能实现精准、细粒度的推理。为此，我们提出START（Spatial and Textual learning for chART understanding），引入（i）图表元素定位和（ii）图表到代码生成，增强MLLM对图表视觉布局与数据细节的理解。为支持空间与文本学习，我们构建了START-Dataset，采用新颖的数据生成流程：先用MLLM将真实图表图像转化为可执行的图表代码，恢复底层数据表示并保留真实图表的视觉分布；再通过大语言模型（LLM）演化代码，精确标注图表元素位置，解决现有方法难以处理的挑战。为评估模型对图表空间结构的理解能力，我们提出图表空间理解基准CS-Bench，填补了全面图表理解评估的空白。实验表明，利用空间与文本学习，START在不同模型规模和基准上均持续优于基线模型，并以明显优势超越先前最先进方法。代码、数据与模型将公开可用。
原文摘要 · Abstract (English)
Chart understanding is crucial for deploying multimodal large language models (MLLMs) in real-world scenarios such as analyzing scientific papers and technical reports. Unlike natural images, charts pair a structured visual layout (spatial property) with an underlying data representation (textual property) -- grasping both is essential for precise, fine-grained chart reasoning. Motivated by this observation, we propose START, the Spatial and Textual learning for chART understanding. Specifically, we introduce (i) chart-element grounding and (ii) chart-to-code generation to strengthen an MLLM's understanding of both chart visual layout and data details. To facilitate spatial and textual learning, we propose the START-Dataset generated with a novel data-generation pipeline that first leverages an MLLM to translate real chart images into executable chart code, recovering the underlying data representation while preserving the visual distribution of real-world charts. We then evolve the code with a Large Language Model (LLM) to ascertain the positions of chart elements that capture the chart's visual structure, addressing challenges that existing methods cannot handle. To evaluate a model's ability to understand chart spatial structures, we propose the Chart Spatial understanding Benchmark (CS-Bench), filling a critical gap in comprehensive chart understanding evaluation. Leveraging spatial and textual learning, START delivers consistent gains across model sizes and benchmarks over the base models and surpasses prior state-of-the-art by a clear margin. Code, data and models will be publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。