arXiv:2412.20715cs.MMcs.CL2024-12被引 7

用轻量模块让大模型更懂图表,生成更准确的自然语言摘要。

ChartAdapter: Large Vision-Language Model for Chart Summarization

  • 设计可学习查询向量与跨模态对齐投影,增强图表到文本的语义对齐。
  • 在19万张图表数据上训练,显著优于现有方法,生成质量更高。
  • 适合需要自动化图表解读的科研、商业分析场景,尤其关注数据可视化。

图表摘要旨在从图表中提取关键信息并以自然语言形式进行解释,对实现高效、易懂的数据分析至关重要。传统方法多采用多阶段流水线,可能导致视觉与文本语义对齐不佳。而现有基于大语言模型的方法往往依赖通用图像或语言能力,忽视图表数据的独特性与挑战。为此,我们提出ChartAdapter——一种轻量级Transformer模块,用于弥合图表与文本摘要之间的差距。该模块通过可学习查询向量提取图表隐含语义,并引入跨模态对齐投影器,强化视觉到语言的生成学习。将ChartAdapter集成至大语言模型后,支持端到端训练与高效摘要生成。为进一步提升训练效果,我们设计了三阶段分层训练流程,并构建了一个大规模专用数据集,包含190,618个样本。在标准Chart-to-Text测试集上的实验表明,本方法显著超越现有方法,包括当前最先进模型,生成高质量摘要。消融实验证明了核心组件的有效性。该工作展示了定制化大模型方法在推进图表理解方面的潜力,为后续研究奠定了坚实基础。

原文摘要 · Abstract (English)

Chart summarization, which focuses on extracting key information from charts and interpreting it in natural language, is crucial for generating and delivering insights through effective and accessible data analysis. Traditional methods for chart understanding and summarization often rely on multi-stage pipelines, which may produce suboptimal semantic alignment between visual and textual information. In comparison, recently developed LLM-based methods are more dependent on the capability of foundation images or languages, while ignoring the characteristics of chart data and its relevant challenges. To address these limitations, we propose ChartAdapter, a novel lightweight transformer module designed to bridge the gap between charts and textual summaries. ChartAdapter employs learnable query vectors to extract implicit semantics from chart data and incorporates a cross-modal alignment projector to enhance vision-to-language generative learning. By integrating ChartAdapter with an LLM, we enable end-to-end training and efficient chart summarization. To further enhance the training, we introduce a three-stage hierarchical training procedure and develop a large-scale dataset specifically curated for chart summarization, comprising 190,618 samples. Experimental results on the standard Chart-to-Text testing set demonstrate that our approach significantly outperforms existing methods, including state-of-the-art models, in generating high-quality chart summaries. Ablation studies further validate the effectiveness of key components in ChartAdapter. This work highlights the potential of tailored LLM-based approaches to advance chart understanding and sets a strong foundation for future research in this area.

图表生成视觉语言模型大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。