ChartScope提升多类型图表理解能力，解决数据对齐与泛化难题。
In-Depth and In-Breadth: Pre-training Multimodal Language Models Customized for Comprehensive Chart Understanding
- 构建多样化图表配对数据生成管道，覆盖多种图表类型。
- 提出双路径训练策略，兼顾数据细节捕捉与推理能力。
- 新基准ChartDQA评估问答与底层数据理解，适合图表分析研究者。
针对科学图表理解中大型视觉语言模型（LVLM）定制方法的局限性，现有方法受限于少数图表类型的成对数据，且缺乏对图表数据对齐的针对性预训练，影响模型对底层数据的理解。本文提出ChartScope，一种面向跨多种图表类型深入理解的优化型LVLM。我们设计了一种高效的数据生成流程，合成涵盖广泛图表类型的配对数据；提出创新的双路径训练策略，使模型在捕捉关键数据特征的同时保持强推理能力。此外，我们构建了ChartDQA新基准，用于评估不同层次的问题回答及底层数据理解能力。实验表明，ChartScope在多种图表类型上显著提升了理解性能。代码与数据已公开：https://davidhalladay.github.io/chartscope_demo。
原文摘要 · Abstract (English)
Recent methods for customizing Large Vision Language Models (LVLMs) for domain-specific tasks have shown promising results in scientific chart comprehension. However, existing approaches face two major limitations: First, they rely on paired data from only a few chart types, limiting generalization to wide range of chart types. Secondly, they lack targeted pre-training for chart-data alignment, which hampers the model's understanding of underlying data. In this paper, we introduce ChartScope, an LVLM optimized for in-depth chart comprehension across diverse chart types. We propose an efficient data generation pipeline that synthesizes paired data for a wide range of chart types, along with a novel Dual-Path training strategy that enabling the model to succinctly capture essential data details while preserving robust reasoning capabilities by incorporating reasoning over the underlying data. Lastly, we establish ChartDQA, a new benchmark for evaluating not only question-answering at different levels but also underlying data understanding. Experimental results demonstrate that ChartScope significantly enhances comprehension on a wide range of chart types. The code and data are available at https://davidhalladay.github.io/chartscope_demo.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。