构建可视化理解数据集SimVecVis，提升大模型对图表的结构解析能力。
SimVecVis: A Dataset for Enhancing MLLMs in Visualization Understanding
- 提出简化向量格式SimVec，编码图表元素的位置、形状与大小信息。
- 在数据驱动问答任务中，使用该数据集微调后模型准确率显著提升。
- 适合研究图表理解、多模态模型训练的研究者使用。
当前多模态大语言模型（MLLMs）在自然图像理解方面表现良好，但在可视化理解上存在瓶颈，主要因其无法解码数据到视觉的映射关系并提取结构化信息。为此，我们提出SimVec，一种新的简化向量格式，用于编码图表中的标记类型、位置和大小等元素。通过将图表信息从SimVec重建，验证了其有效性。进而构建新数据集SimVecVis，包含三方面内容：图表位图图像、对应的SimVec表示以及以数据为中心的问答对（QA）与解释性思维链（CoT）描述。我们使用不同维度的数据集对先进MLLMs（如MiniCPM和Qwen-VL）进行微调，实验表明，该数据集显著提升了具备良好空间感知能力的模型（如MiniCPM）在数据驱动问答任务中的性能。相关数据集与源代码已开源：https://github.com/VIDA-Lab/SimVecVis。
原文摘要 · Abstract (English)
Current multimodal large language models (MLLMs), while effective in natural image understanding, struggle with visualization understanding due to their inability to decode the data-to-visual mapping and extract structured information. To address these challenges, we propose SimVec, a novel simplified vector format that encodes chart elements such as mark type, position, and size. The effectiveness of SimVec is demonstrated by using MLLMs to reconstruct chart information from SimVec formats. Then, we build a new visualization dataset, SimVecVis, to enhance the performance of MLLMs in visualization understanding, which consists of three key dimensions: bitmap images of charts, their SimVec representations, and corresponding data-centric question-answering (QA) pairs with explanatory chain-of-thought (CoT) descriptions. We finetune state-of-the-art MLLMs (e.g., MiniCPM and Qwen-VL), using SimVecVis with different dataset dimensions. The experimental results show that it leads to substantial performance improvements of MLLMs with good spatial perception capabilities (e.g., MiniCPM) in data-centric QA tasks. Our dataset and source code are available at: https://github.com/VIDA-Lab/SimVecVis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。