用图结构增强多模态推理,让大模型更懂图像文字间关系。
Mario: Multimodal Graph Reasoning with Large Language Models
- 基于图拓扑优化图文特征,细粒度对齐跨模态信息。
- 在多个数据集上超越现有模型,零样本和有监督任务均领先。
- 适合做多模态图推理、知识图谱融合的研究者使用。
大语言模型(LLM)的发展为多模态推理带来了新可能。然而,现有方法大多依赖预训练视觉-语言模型(VLM)孤立编码图文对,忽视了真实世界多模态数据天然形成的关联结构。为此,我们提出Mario,一个统一框架,支持在异构多模态图(MMG)上进行基于大模型的推理。每个节点包含文本与视觉属性,边提供结构线索。Mario解决两大挑战:跨模态一致性弱、异构模态偏好问题。首先,设计图条件下的VLM,通过图拓扑引导的细粒度跨模态对比学习,联合优化图文特征。其次,采用模态自适应图指令微调机制,将对齐后的多模态特征组织成图感知指令视图,并利用可学习路由器为每个节点及其邻域选择最有效的模态配置输入LLM。在多个多模态图基准测试中,Mario在节点分类与链接预测任务上,无论有监督还是零样本场景,均持续优于当前最优模型。代码将在https://github.com/sunyuanfu/Mario公开。
原文摘要 · Abstract (English)
Recent advances in large language models (LLMs) have opened new avenues for multimodal reasoning. Yet, most existing methods still rely on pretrained vision-language models (VLMs) to encode image-text pairs in isolation, ignoring the relational structure that real-world multimodal data naturally form. This motivates reasoning on multimodal graphs (MMGs), where each node has textual and visual attributes and edges provide structural cues. Enabling LLM-based reasoning on such heterogeneous multimodal signals while preserving graph topology introduces two key challenges: resolving weak cross-modal consistency and handling heterogeneous modality preference. To address this, we propose Mario, a unified framework that simultaneously resolves the two above challenges and enables effective LLM-based reasoning over MMGs. Mario consists of two innovative stages. Firstly, a graph-conditioned VLM design that jointly refines textual and visual features through fine-grained cross-modal contrastive learning guided by graph topology. Secondly, a modality-adaptive graph instruction tuning mechanism that organizes aligned multimodal features into graph-aware instruction views and employs a learnable router to surface, for each node and its neighborhood, the most informative modality configuration to the LLM. Extensive experiments across diverse MMG benchmarks demonstrate that Mario consistently outperforms state-of-the-art graph models in both supervised and zero-shot scenarios for node classification and link prediction. The code will be made available at https://github.com/sunyuanfu/Mario.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。