arXiv:2607.15687cs.LG2026-07

联邦学习下多模态图模型的对齐框架,保护隐私同时提升性能。

Toward Federated Multimodal Graph Foundation Models: A Topology-Aware Multimodal Alignment Framework

论文配图:Toward Federated Multimodal Graph Foundation Models: A Topology-Aware Multimodal Alignment Framework
图 1 · 摘自论文原文
  • 分两阶段训练:预训练时通过最优传输对齐跨模态语义,融合拓扑结构。
  • 在12个数据集上比基线最高提升12.96%,少样本场景下仍领先5.71%。
  • 适合需要隐私保护的多源多模态图学习任务,如医疗、电商联合建模。

多模态属性图(MAGs)在社交平台、电商和生物医学网络中广泛应用,其节点包含图像、文本等多模态信息及拓扑结构,提供比单模态图更丰富的语义信号。然而,这些图数据分散于受隐私限制的多个独立机构中,难以直接共享。为在不暴露原始数据的前提下实现可迁移模型训练,需结合多模态图学习与联邦学习。现有方法仅覆盖其中一端。为此,本文提出FedGAMMA,将联邦多模态图基础模型学习建模为两阶段的语义-结构对齐问题:预训练阶段,共享-私有语义增强器通过最优传输分离跨模态共性与模态特异性;拓扑感知图融合模块利用语义残差图与双位置编码解耦语义与结构视图;双通道亲和度感知聚合机制基于特征与图中心点估算客户端相似性,无需暴露原始数据。微调阶段,通过轻量级图感知提示、可控探索的共享提示池及通道级提示同步,适配预训练编码器。在12个多模态图数据集上的实验表明,FedGAMMA在下游任务中持续超越多种基线,性能提升最高达12.96%;在多领域数据集上,少样本场景下相比竞争方法最高提升5.71%。

原文摘要 · Abstract (English)

Multimodal-attributed graphs (MAGs), whose nodes carry modalities such as images and text alongside topological structure, now pervade applications including social platforms, e-commerce, and biomedical networks, offering richer semantic signals than single-modality graphs. In practice, such graphs are fragmented across privacy-restricted silos owned by different platforms and institutions, so learning a broadly transferable model over them demands collaborative training that never exposes raw data. This places the task at the intersection of multimodal graph learning and federated learning, yet existing methods cover only one side of it. To address the challenges from these two perspectives, we propose FedGAMMA, casting federated multimodal graph foundation learning as a two-stage semantic-structural alignment problem of federated pre-training and prompt-based fine-tuning. During pre-training, a shared-private semantic enhancer disentangles cross-modal commonality from modality-specific information, aligning it through optimal transport, a topology-aware graph fusion module decouples semantic and structural views via semantic residual graphs and dual positional encodings, and a dual-channel affinity-aware aggregation mechanism estimates client similarity from feature and graph centroids without exposing raw data. During fine-tuning, FedGAMMA adapts the pretrained encoder through lightweight graph-aware prompts, a shared prompt pool with controlled exploration, and channel-wise prompt synchronization. Experiments on twelve multimodal graph datasets show FedGAMMA consistently surpassing a broad range of baselines across downstream tasks, with gains of up to 12.96%. FedGAMMA further outperforms competitive baselines accross multi-domain datasets on multiple tasks with up to 5.71% under few-shot learning scenario.

联邦学习多模态图隐私保护基础模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。