一个模型搞定不同模态的图数据,通用性强。
One Model, Many Graphs: Learning over Attributed Graphs across Heterogeneous Modalities with Vision-Language Models

- 用预训练视觉语言模型做统一骨架,适配多种图数据
- 在多类图任务上超越现有GNN和LLM模型表现
- 支持文本、图像或混合属性图,跨图泛化能力强
视觉语言模型(VLM)为文本与视觉信息提供了统一表征空间,但其作为图结构数据通用骨干的潜力尚未被充分挖掘。实际中,属性图存在显著的模态异质性:部分图仅有文本节点属性,部分仅有视觉属性,还有些同时包含两者。现有图学习方法通常针对固定模态架构设计,需为不同场景单独建模,限制了可扩展性与跨图泛化能力。为此,我们提出OMG-VLM(One Model, Many Graphs with Vision-Language Models),一种面向异构模态架构的属性图统一学习框架。OMG-VLM利用预训练VLM作为共享骨干,并引入结构感知图适配器,在保留与VLM原生嵌入空间兼容性的前提下融合邻域信息。该设计使单一模型能有效处理纯文本、纯图像及多模态属性图。在多个领域的广泛实验表明,OMG-VLM在节点分类与链接预测等任务上持续优于最先进的GNN与LLM基线模型,且对未见图与不同模态架构展现出强泛化能力。
原文摘要 · Abstract (English)
Vision-language models (VLMs) provide a unified representation space for textual and visual information, yet their potential as general-purpose backbones for graph-structured data remains largely unexplored. In practice, attributed graphs exhibit substantial modality heterogeneity: some graphs contain only textual node attributes, others only visual attributes, while still others provide both. Existing graph learning approaches are typically designed for fixed modality schemas, requiring separate models for different settings and limiting scalability and cross-graph generalization. To bridge this gap, we present OMG-VLM (One Model, Many Graphs with Vision-Language Models), a unified framework for learning over attributed graphs across heterogeneous modality schemas. OMG-VLM leverages a pretrained VLM as a shared backbone and introduces structure-aware graph adapters that integrate neighborhood information while remaining compatible with the VLM's native embedding space. This design enables effective learning over text-attributed, image-attributed, and multi-attributed graphs within a single model. Extensive experiments across diverse domains show that OMG-VLM consistently outperforms state-of-the-art GNN- and LLM-based baselines on attributed graph learning tasks such as node classification and link prediction, while exhibiting strong generalization to unseen graphs and varying modality schemas. The source code is available at https://github.com/Jo-eyang/OMG-VLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。