解决联邦图学习中缺失模态的问题,提升不完整数据的协作建模效果。
PRISM: Topology-Aware Cross-Modal Imputation for Modality-Deficient Federated Graph Learning

- 通过联邦协作检索并注入缺失模态语义,结合图结构控制信息传播。
- 在六大数据集上平均性能优于现有方法4.48%。
- 适合处理实际中存在模态缺失的分布式图学习场景。
多模态联邦图学习(MM-FGL)旨在从包含文本与图像的分布式图中协同学习。然而现实中的客户端可能缺乏共同的模态基础:视觉搜索客户端仅有图像-交互图而无卖家描述,目录类客户端则提供文本但无产品图像。我们称此为客户端级模态缺失。不同于随机实例缺失,缺失客户端缺乏重建缺失模态所需的本地语义基础。更重要的是,在图学习中,不完整的表示会初始化消息传递,导致插补错误被接收端的拓扑结构过滤、混合并放大。为此,我们提出PRISM(Proactive Retrieval and Imputation via Structural Meta-prompting),一种拓扑感知的联邦跨模态插补框架。不同于仅依赖本地观测重建缺失模态,PRISM从联邦中恢复缺失模态语义,并在拓扑感知控制下引入本地图传播。在六个多模态图数据集上的实验表明,PRISM在以图为中心和以模态为中心的任务中均能持续提升模态缺失客户端的表现,平均优于最先进基线4.48%。
原文摘要 · Abstract (English)
Multimodal federated graph learning (MM-FGL) aims to collaboratively learn from decentralized graphs with text and images. However, real-world clients may not share a common modality basis: a visual-search client may contain image--interaction graphs but no seller descriptions, while a catalog client may provide text but no product images. We refer to this practical setting as client-level modality deficiency. Unlike random instance-wise missingness, a deficient client lacks the local semantic basis needed to reconstruct the absent modality. More importantly, in graph learning, incomplete representations initialize message passing, so imputation errors can be filtered, mixed, and amplified by the receiving topology. To address this gap, we propose \textbf{PRISM} (\textbf{P}roactive \textbf{R}etrieval and \textbf{I}mputation via \textbf{S}tructural \textbf{M}eta-prompting), a topology-aware federated cross-modal imputation framework. Rather than reconstructing the missing modality solely from local observations, PRISM recovers missing-modality semantics from the federation and introduces them into local graph propagation under topology-aware control. Experiments on six multimodal graph datasets across graph-centric and modality-centric tasks show that PRISM consistently improves modality-deficient clients, outperforming state-of-the-art baselines by \textbf{4.48}\% on average.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。