用结构化表格提升金融图表问答的准确率
Enhancing Financial VQA in Vision Language Models using Intermediate Structured Representations
- 将图表转为线性表格作为中间表示
- 数值理解准确率提升18.7%，分类匹配更精准
- 适合需要高精度图表分析的金融场景
图表解读对视觉数据分析至关重要，但自动模型准确提取图表信息仍具挑战。本研究在自建的5万张柱状图数据集上微调DEPLOT模型，该模型可将图表图像转换为线性表格。数据集包含简单、堆叠及分组柱状图，针对其结构特征设计。微调后的DEPLOT在1000张测试图上评估，采用相对映射相似度（RMS）和相对数值集合相似度（RNSS）两个指标。为进一步探究大语言模型的推理能力，另构建100张图表与问答对。结果表明，提供结构化中间表格能显著提升大语言模型的推理表现，优于直接使用图像输入。
原文摘要 · Abstract (English)
Chart interpretation is crucial for visual data analysis, but accurately extracting information from charts poses significant challenges for automated models. This study investigates the fine-tuning of DEPLOT, a modality conversion module that translates the image of a plot or chart to a linearized table, on a custom dataset of 50,000 bar charts. The dataset comprises simple, stacked, and grouped bar charts, targeting the unique structural features of these visualizations. The finetuned DEPLOT model is evaluated against its base version using a test set of 1,000 images and two metrics: Relative Mapping Similarity (RMS), which measures categorical mapping accuracy, and Relative Number Set Similarity (RNSS), which evaluates numerical interpretation accuracy. To further explore the reasoning capabilities of large language models (LLMs), we curate an additional set of 100 bar chart images paired with question answer sets. Our findings demonstrate that providing a structured intermediate table alongside the image significantly enhances LLM reasoning performance compared to direct image queries.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。