提出BRAIN模型,统一建模多模态图的跨模态与跨范围关系。
Towards Unified Multimodal Graph Foundation Model: A Bridge-Router-Adapter Based Approach

- 用分层路由机制按邻域范围动态分配模态权重。
- 在9个数据集上提升节点分类准确率最高4.73%。
- 适合需要跨模态图推理的任务,如图文关联预测。
多模态图将不同模态(如文本和图像)的节点属性与关系结构耦合,支持拓扑结构与跨模态属性联合建模。多模态图基础模型旨在从这类数据中学习可迁移的统一表示,适用于多种图领域与下游任务。然而现有方法存在两大局限:(1) 跨范围上下文纠缠,将局部到全局的图上下文混合为统一表示,掩盖了其范围差异;(2) 模态路由忽视范围变化,固定在特定邻域内进行模态选择。为此,我们提出BRAIN模型,聚焦于结合邻域范围与模态组合的图上下文。BRAIN包含:条件式桥接模块(Bridge),融合跨越局部到全局邻域范围的结构信息与不同模态组合;分层路由模块(Router),评估范围与任务的相关性,并在各范围内独立选择模态组合,使模态效用随图范围变化;轻量残差适配器(Adapter),进一步针对下游任务优化路由后的嵌入表示。BRAIN通过多图预训练+任务微调方式训练。在九个数据集和四类任务上的实验表明,其在节点分类与链接预测上相对最强基线最高提升4.73%,在四类图到文本与两类图到图像任务上平均相对提升达14.72%。
原文摘要 · Abstract (English)
Multimodal graphs couple node attributes in different modalities, such as text and images, with relational structure, enabling topological structure and cross-modality attributes to be modeled jointly. Multimodal graph foundation models seek unified representations from such data that transfer across different graph domains and downstream tasks. However, existing methods exhibit two fundamental limitations. (1) Cross-Scope Context Entanglement. They merge scope-specific graph contexts into a unified representation, obscuring their distinctions during multimodal construction. (2) Scope-Ignorant Modality Routing. They route modalities within a fixed graph scope, overlooking how modality relevance varies across neighborhood ranges. To address these challenges, we propose BRAIN, a unified model that focuses on graph context that combines neighborhood scope with modality composition. BRAIN comprises a scope-conditioned Bridge that combines structural information spanning local-to-global neighborhood scopes with different modality compositions; a hierarchical Router that estimates the relevance between the scope and the task, and selects compositions separately within each scope, allowing modality utility to vary with graph range; and a lightweight residual Adapter that further specializes the routed embedding for downstream prediction. BRAIN is trained through multi-graph pretraining followed by task-specific adaptation. Experiments across nine datasets and four task families demonstrate its broad effectiveness, improving node-classification and link-prediction performance by up to 4.73% relative to the strongest baseline, while achieving an average relative improvement of 14.72% across four graph-to-text and two graph-to-image metrics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。