提出多模态图模型CHARM,实现跨域零样本迁移
CHARM: A Multimodal Graph Foundation Model with Hierarchical Context Modeling for Zero-Shot Transfer

- 用分层上下文建模替代原始节点,融合多模态语义
- 零样本下在多个图任务上性能超越现有方法
- 适合需要跨域迁移的多模态图分析场景
图基础模型(GFMs)为跨图领域和任务的知识迁移提供了新范式。现实世界中的图常关联文本、图像等多模态信息,使多模态图成为表示复杂实体与关系的关键。然而,为每个新图领域收集标签并进行模型适配成本高昂且不可行,推动了零样本迁移的需求。目前,多模态图上的零样本迁移仍缺乏研究。现有基于GNN的图基础模型通常需下游微调,而基于LLM的方法主要针对单模态图或单一领域任务。该场景面临两大挑战:一是模型需从各模态中泛化知识,同时捕捉可迁移的跨模态关系;二是缺乏目标域微调时,节点表征易受领域特异性结构和模态特征干扰,掩盖未见领域的共享概念。为此,我们提出CHARM——一种具有分层上下文建模的多模态图基础模型,支持零样本迁移。CHARM将孤立的原始节点替换为捕获多模态语义与跨模态关系的分层图上下文,将领域特异的节点模式映射到共享的高层概念,降低对目标域监督或适应的依赖。一个模态感知的图上下文编码器融合多模态信息与图结构,并将表征转化为大语言模型的图令牌。实验表明,CHARM在零样本多模态图任务上持续提升性能。
原文摘要 · Abstract (English)
Graph foundation models (GFMs) have emerged as a promising paradigm for transferring knowledge across graph domains and tasks. Real-world graphs associate nodes with text, images, and other modalities, making multimodal graphs essential for representing complex entities and relations. Moreover, collecting labels and adapting models for every new graph domain is costly and often infeasible, motivating zero-shot transfer. Unfortunately, zero-shot transfer on multimodal graphs remains underexplored. Existing GNN-based graph foundation models typically require downstream adaptation, whereas LLM-based graph methods mainly address unimodal graphs or tasks within a single domain. This setting presents two key challenges. First, models must generalize knowledge from individual modalities while capturing transferable cross-modal relations. Second, without target-domain fine-tuning, node representations remain entangled with domain-specific structures and modality-specific characteristics, obscuring shared concepts in unseen domains. To address these challenges, we propose CHARM, a multimodal graph foundation model with hierarchical context modeling for zero-shot transfer. CHARM replaces isolated raw nodes with hierarchical graph contexts that capture multimodal semantics and cross-modal relations. These contexts map domain-specific node patterns to shared high-level concepts, reducing reliance on target-domain supervision or adaptation. A modality-aware graph context encoder integrates multimodal information with graph structure and converts the resulting representations into graph tokens for a large language model . Experiments show consistent improvements on zero-shot multimodal graph tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。