arXiv:2603.13370cs.CVcs.LG2026-03被引 1

用视觉语言模型提升多模态图学习,验证三类融合方式效果

GraphVLM: Benchmarking Vision Language Models for Multimodal Graph Learning

  • 将VLM作为编码器、对齐器或预测器,融入图学习框架
  • 在6个数据集上验证,VLM作为预测器效果最佳
  • 适合研究多模态图推理与大模型融合的学者

视觉语言模型(VLM)在对齐和理解多模态信号方面表现卓越,但其在结构化数据上的推理能力——即多模态实体通过显式关系图连接——仍鲜被探索。这在社交网络、推荐系统和科学发现等真实场景中至关重要。为此,我们提出GraphVLM,一个系统性基准,用于评估和利用VLM在多模态图学习(MMGL)中的潜力。该基准考察三种互补的融合范式:(1) VLM-as-Encoder,通过多模态特征融合增强图神经网络;(2) VLM-as-Aligner,在潜在空间或语言空间中桥接模态,支持基于大语言模型的结构化推理;(3) VLM-as-Predictor,直接以VLM作为多模态图学习任务的主干模型。在六个来自不同领域的数据集上进行广泛实验表明,三种范式均能提升多模态图学习性能。其中,VLM-as-Predictor展现出最显著且一致的性能提升,揭示了视觉语言模型作为多模态图学习新基础的巨大潜力。基准代码已公开于https://github.com/oamyjin/GraphVLM。

原文摘要 · Abstract (English)

Vision-Language Models (VLMs) have demonstrated remarkable capabilities in aligning and understanding multimodal signals, yet their potential to reason over structured data, where multimodal entities are connected through explicit relational graphs, remains largely underexplored. Unlocking this capability is crucial for real-world applications such as social networks, recommendation systems, and scientific discovery, where multimodal information is inherently structured. To bridge this gap, we present GraphVLM, a systematic benchmark designed to evaluate and harness the capabilities of VLMs for multimodal graph learning (MMGL). GraphVLM investigates three complementary paradigms for integrating VLMs with graph reasoning: (1) VLM-as-Encoder, which enriches graph neural networks through multimodal feature fusion; (2) VLM-as-Aligner, which bridges modalities in latent or linguistic space to facilitate LLM-based structured reasoning; and (3) VLM-as-Predictor, which directly employs VLMs as multimodal backbones for graph learning tasks. Extensive experiments across six datasets from diverse domains demonstrate that VLMs enhance multimodal graph learning via all three roles. Among these paradigms, VLM-as-Predictor achieves the most substantial and consistent performance gains, revealing the untapped potential of vision-language models as a new foundation for multimodal graph learning. The benchmark code is publicly available at https://github.com/oamyjin/GraphVLM.

多模态图学习视觉语言模型图神经网络跨模态对齐

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。