用AI解析地铁图并回答问题，让机器理解图像中的复杂结构。
Visual Graph Question Answering with ASP and LLMs for Language Parsing
- 结合视觉识别与大模型语言能力，用逻辑编程推理图像信息。
- 在新构建的地铁图数据集上达到73%平均准确率。
- 无需额外训练，适合需要可解释性的智能问答场景。
视觉问答（VQA）需处理多模态输入，答案集编程（ASP）在提升模块化VQA系统的可解释性方面展现出巨大潜力。本文研究如何将ASP与视觉及自然语言处理模块结合，解决一种新型且具有挑战性的VQA任务——图像中的图结构（非符号形式）。这类图像广泛用于可视化，尤其以交通网络为例。我们引入一个新数据集，扩展现有数据集，加入类似地铁线路的图像。所提模块化神经符号方法整合了光学图识别、预训练OCR网络解析标签、大语言模型（LLMs）进行语言处理，以及ASP进行推理。该方法作为首个基线，在数据集上实现73%的整体平均准确率。评估进一步证明，无需额外训练的预训练模型与逻辑编程结合，可有效解决复杂的视觉问答任务。
原文摘要 · Abstract (English)
Visual Question Answering (VQA) is a challenging problem that requires to process multimodal input. Answer-Set Programming (ASP) has shown great potential in this regard to add interpretability and explainability to modular VQA architectures. In this work, we address the problem of how to integrate ASP with modules for vision and natural language processing to solve a new and demanding VQA variant that is concerned with images of graphs (not graphs in symbolic form). Images containing graph-based structures are an ubiquitous and popular form of visualisation. Here, we deal with the particular problem of graphs inspired by transit networks, and we introduce a novel dataset that amends an existing one by adding images of graphs that resemble metro lines. Our modular neuro-symbolic approach combines optical graph recognition for graph parsing, a pretrained optical character recognition neural network for parsing labels, Large Language Models (LLMs) for language processing, and ASP for reasoning. This method serves as a first baseline and achieves an overall average accuracy of 73% on the dataset. Our evaluation provides further evidence of the potential of modular neuro-symbolic systems, in particular with pretrained models that do not involve any further training and logic programming for reasoning, to solve complex VQA tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。