探索视觉化图数据如何提升多模态大模型的图结构理解能力
Exploring Graph Structure Comprehension Ability of Multimodal Large Language Models: Case Studies
- 用图文并茂方式让大模型理解图结构，对比纯文本表示效果
- 在节点、边、图三级任务上验证视觉信息提升模型表现
- 揭示视觉模态优势与局限，适合图神经网络研究者参考
大型语言模型（LLMs）在处理多种数据结构方面表现出色，包括图结构。尽管以往研究侧重于图的文本编码方法，但多模态大模型的出现为图理解开辟了新方向。这些能够处理文本和图像的先进模型，通过结合视觉表示与传统文本数据，有望提升图理解能力。本研究在节点、边和图三个层级上，考察了图可视化对多模态模型性能的影响，并对比了多模态方法与纯文本图表示的有效性。实验结果揭示了利用视觉图模态增强大模型图结构理解能力的潜力与局限。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have shown remarkable capabilities in processing various data structures, including graphs. While previous research has focused on developing textual encoding methods for graph representation, the emergence of multimodal LLMs presents a new frontier for graph comprehension. These advanced models, capable of processing both text and images, offer potential improvements in graph understanding by incorporating visual representations alongside traditional textual data. This study investigates the impact of graph visualisations on LLM performance across a range of benchmark tasks at node, edge, and graph levels. Our experiments compare the effectiveness of multimodal approaches against purely textual graph representations. The results provide valuable insights into both the potential and limitations of leveraging visual graph modalities to enhance LLMs' graph structure comprehension abilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。