arXiv:2506.10282cs.LG2025-06被引 8

构建首个多模态图学习统一评测基准,验证多模态融合提升图学习效果。

Graph-MLLM: Harnessing Multimodal Large Language Models for Multimodal Graph Learning

  • 设计Graph-MLLM基准,系统评估三种多模态图学习范式
  • 发现图文联合建模显著提升性能,文本描述优于直接视觉输入
  • 微调大模型在多数场景超越现有方法,无需显式图结构信息

多模态大语言模型(MLLM)在跨模态表示与理解方面表现卓越,但通常仅关注成对模态对齐,忽视数据点间的结构关系。将多模态信息与图结构结合(即多模态图,MMG)对社交网络、医疗和推荐系统等真实场景至关重要。现有方法可分为三类:编码器型(增强GNN)、对齐型(实现基于语言模型的图推理)、预测型(以MLLM为独立推理器)。尽管进展显著,该领域缺乏统一评测基准,难以公平比较。为此,我们提出Graph-MLLM,一个涵盖六个不同领域数据集的综合性基准,系统评估上述三类范式。实验表明,联合使用节点的视觉与文本属性能显著提升图学习效果,即使采用预训练的图文对齐模型(如CLIP)作为编码器亦然;进一步将视觉属性转为文本描述,性能优于直接使用视觉输入。此外,对特定多模态图微调MLLM可取得多数场景下的最先进结果,甚至无需显式图结构信息。我们开源该工具库,旨在推动快速、公平的评估并激发更多创新研究。

原文摘要 · Abstract (English)

Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities in representing and understanding diverse modalities. However, they typically focus on modality alignment in a pairwise manner while overlooking structural relationships across data points. Integrating multimodality with structured graph information (i.e., multimodal graphs, MMGs) is essential for real-world applications such as social networks, healthcare, and recommendation systems. Existing MMG learning methods fall into three paradigms based on how they leverage MLLMs: Encoder, Aligner, and Predictor. MLLM-as-Encoder focuses on enhancing graph neural networks (GNNs) via multimodal feature fusion; MLLM-as-Aligner aligns multimodal attributes in language or hidden space to enable LLM-based graph reasoning; MLLM-as-Predictor treats MLLMs as standalone reasoners with in-context learning or fine-tuning. Despite their advances, the MMG field lacks a unified benchmark to fairly evaluate across these approaches, making it unclear what progress has been made. To bridge this gap, we present Graph-MLLM, a comprehensive benchmark for multimodal graph learning by systematically evaluating these three paradigms across six datasets with different domains. Through extensive experiments, we observe that jointly considering the visual and textual attributes of the nodes benefits graph learning, even when using pre-trained text-to-image alignment models (e.g., CLIP) as encoders. We also find that converting visual attributes into textual descriptions further improves performance compared to directly using visual inputs. Moreover, we observe that fine-tuning MLLMs on specific MMGs can achieve state-of-the-art results in most scenarios, even without explicit graph structure information. We hope that our open-sourced library will facilitate rapid, equitable evaluation and inspire further innovative research in this field.

多模态图大模型图学习评测基准

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。