arXiv:2510.18825cs.CV2025-10NeurIPS被引 1

用分层掩码统一图注意力机制,提升模型表现与灵活性。

Unifying and Enhancing Graph Transformers via a Hierarchical Mask Framework

  • 提出分层掩码框架,将架构设计转化为掩码构造,统一建模思路。
  • 理论证明接收域越大、标签一致性越高,分类准确率越高。
  • 引入M3Dphormer模型,融合多层级掩码与双模式注意力,性能领先。

图注意力网络(GTs)因其能建模多样节点交互而成为图表示学习的强大范式。然而,现有方法常依赖针对特定交互的复杂架构,限制了灵活性。为此,我们提出统一的分层掩码框架,揭示了模型架构与注意力掩码构造之间的内在等价性。该框架通过精心设计的注意力掩码,一致地捕捉多样化交互。理论分析表明,正确分类概率与感受野大小和标签一致性正相关,由此得出根本设计原则:有效注意力掩码应具备足够大的感受野和高标签一致性。尽管单一掩码无法在所有场景下满足该原则,但分层掩码具有互补优势,促使它们的有效整合。随后,我们提出M3Dphormer——一种基于专家混合的图变压器,包含多层级掩码与双注意力计算。该模型融合三种理论指导的分层掩码,并采用双层专家路由机制自适应整合多层级交互信息。为保证可扩展性,进一步引入双注意力计算方案,根据局部掩码稀疏度动态切换稠密与稀疏模式。在多个基准测试上的大量实验表明,M3Dphormer达到最先进性能,验证了统一框架与模型设计的有效性。

原文摘要 · Abstract (English)

Graph Transformers (GTs) have emerged as a powerful paradigm for graph representation learning due to their ability to model diverse node interactions. However, existing GTs often rely on intricate architectural designs tailored to specific interactions, limiting their flexibility. To address this, we propose a unified hierarchical mask framework that reveals an underlying equivalence between model architecture and attention mask construction. This framework enables a consistent modeling paradigm by capturing diverse interactions through carefully designed attention masks. Theoretical analysis under this framework demonstrates that the probability of correct classification positively correlates with the receptive field size and label consistency, leading to a fundamental design principle: an effective attention mask should ensure both a sufficiently large receptive field and a high level of label consistency. While no single existing mask satisfies this principle across all scenarios, our analysis reveals that hierarchical masks offer complementary strengths, motivating their effective integration. Then, we introduce M3Dphormer, a Mixture-of-Experts-based Graph Transformer with Multi-Level Masking and Dual Attention Computation. M3Dphormer incorporates three theoretically grounded hierarchical masks and employs a bi-level expert routing mechanism to adaptively integrate multi-level interaction information. To ensure scalability, we further introduce a dual attention computation scheme that dynamically switches between dense and sparse modes based on local mask sparsity. Extensive experiments across multiple benchmarks demonstrate that M3Dphormer achieves state-of-the-art performance, validating the effectiveness of our unified framework and model design.

图神经网络注意力机制分层掩码M3Dphormer

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。