arXiv:2602.00067cs.LGcs.AI2026-02被引 1

解决多模态图学习中的模态混淆问题,通过节点拆分与图重连提升性能。

Modality as Heterogeneity: Node Splitting and Graph Rewiring for Multimodal Graph Learning

  • 将节点拆分为模态专用部分,用专家网络处理异质信息流
  • 在三个基准上超越强基线,且训练效率优于典型MoE模型
  • 理论分析揭示其能自适应过滤模态子空间,增强泛化能力

多模态图因其强大的表征能力与广泛应用而受到越来越多关注,但其也带来了严重的模态混淆问题。为此,我们提出NSG-MoE框架,结合节点拆分与图重连机制,以及结构化混合专家(MoE)架构。该方法显式地将每个节点分解为模态专用组件,并为关系感知的专家分配以处理异质消息流,从而在保留结构信息和多模态语义的同时,缓解通用GNN中常见的不良混合效应。在三个多模态基准上的大量实验表明,NSG-MoE始终优于强基线。尽管引入了通常计算开销较大的MoE,本方法仍实现了具有竞争力的训练效率。除了实证结果外,我们还进行了谱分析,揭示了NSG对模态专用子空间的自适应滤波作用,解释了其解耦行为;此外,信息论分析显示,NSG的架构约束降低了数据与参数之间的互信息,提升了泛化能力。

原文摘要 · Abstract (English)

Multimodal graphs are gaining increasing attention due to their rich representational power and wide applicability, yet they introduce substantial challenges arising from severe modality confusion. To address this issue, we propose NSG (Node Splitting Graph)-MoE, a multimodal graph learning framework that integrates a node-splitting and graph-rewiring mechanism with a structured Mixture-of-Experts (MoE) architecture. It explicitly decomposes each node into modality-specific components and assigns relation-aware experts to process heterogeneous message flows, thereby preserving structural information and multimodal semantics while mitigating the undesirable mixing effects commonly observed in general-purpose GNNs. Extensive experiments on three multimodal benchmarks demonstrate that NSG-MoE consistently surpasses strong baselines. Despite incorporating MoE -- which is typically computationally heavy -- our method achieves competitive training efficiency. Beyond empirical results, we provide a spectral analysis revealing that NSG performs adaptive filtering over modality-specific subspaces, thus explaining its disentangling behavior. Furthermore, an information-theoretic analysis shows that the architectural constraints imposed by NSG reduces mutual information between data and parameters and improving generalization capability.

多模态图节点拆分MoE图学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。