arXiv:2410.09761cs.AIcs.IR2024-10被引 8

将图表图像转化为包含视觉元素与语义关系的知识图谱,提升数据理解与应用效果。

ChartKG: A Knowledge-Graph-Based Representation for Chart Images

  • 用知识图谱统一建模图表的视觉元素与语义关系。
  • 通过CNN、YOLOv5和OCR识别图表并构建结构化知识图谱。
  • 适用于语义感知的图表检索与问答,适合数据可视化研究者。

由于数据可视化广泛应用,柱状图、饼图、折线图等图表图像正呈爆炸式增长。因此,从图表中挖掘知识对图表检索、知识图谱补全等下游任务至关重要。然而,现有方法多聚焦于将图表转为原始数据,忽视其视觉编码与语义含义,导致信息丢失。本文提出ChartKG,一种基于知识图谱的图表图像表示方法,可统一建模图表中的视觉元素及其语义关系,包括视觉编码与洞察。我们设计了一个通用框架,整合一系列图像处理技术:使用CNN分类图表类型,YOLOv5与OCR解析图表内容,规则方法构建知识图谱。通过四个案例展示该表示如何捕捉图表细节与语义关联,并验证其在语义感知图表检索与图表问答中的优势。定量评估表明,对象识别与OCR两个核心模块具有有效性,支持ChartKG的实用性。

原文摘要 · Abstract (English)

Chart images, such as bar charts, pie charts, and line charts, are explosively produced due to the wide usage of data visualizations. Accordingly, knowledge mining from chart images is becoming increasingly important, which can benefit downstream tasks like chart retrieval and knowledge graph completion. However, existing methods for chart knowledge mining mainly focus on converting chart images into raw data and often ignore their visual encodings and semantic meanings, which can result in information loss for many downstream tasks. In this paper, we propose ChartKG, a novel knowledge graph (KG) based representation for chart images, which can model the visual elements in a chart image and semantic relations among them including visual encodings and visual insights in a unified manner. Further, we develop a general framework to convert chart images to the proposed KG-based representation. It integrates a series of image processing techniques to identify visual elements and relations, e.g., CNNs to classify charts, yolov5 and optical character recognition to parse charts, and rule-based methods to construct graphs. We present four cases to illustrate how our knowledge-graph-based representation can model the detailed visual elements and semantic relations in charts, and further demonstrate how our approach can benefit downstream applications such as semantic-aware chart retrieval and chart question answering. We also conduct quantitative evaluations to assess the two fundamental building blocks of our chart-to-KG framework, i.e., object recognition and optical character recognition. The results provide support for the usefulness and effectiveness of ChartKG.

知识图谱图表理解视觉推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。