arXiv:2505.16130cs.LGcs.AI2025-05NeurIPS被引 15

用生成式Transformer突破图神经网络的表达瓶颈,实现更大规模下的性能提升。

Generative Graph Pattern Machine

  • 将图结构拆解为子结构序列,用生成式预训练学习通用表示
  • 在ogbn-arxiv上模型增至60M参数仍持续提升,优于此前3M即饱和的方法
  • 适合需要大规模图学习和跨任务迁移的场景

图神经网络主要依赖消息传递机制,通过局部邻居聚合迭代更新节点表示。尽管取得成功,消息传递存在表达能力受限、过度平滑、过度压缩及难以建模长程依赖等根本问题,限制了可扩展性:增大数据或模型规模常无法提升性能。为此,我们探索消息传递之外的路径,提出生成式图模式机器(G$^2$PM),一种面向图的生成式Transformer预训练框架。G$^2$PM将图实例(节点、边或整图)表示为子结构序列,并对序列进行生成式预训练以学习可泛化、可迁移的表示。实证表明,G$^2$PM展现出强可扩展性:在ogbn-arxiv基准上,模型规模达60M参数时仍持续提升,优于此前生成方法在约3M参数即达到饱和的表现。此外,我们系统分析了模型设计空间,揭示了促进可扩展性和泛化性的关键架构选择。在节点/边/图分类、迁移学习和跨图预训练等多种任务中,G$^2$PM均持续超越强基线,为可扩展图学习奠定坚实基础。代码与数据集见https://github.com/Zehong-Wang/G2PM。

原文摘要 · Abstract (English)

Graph neural networks (GNNs) have been predominantly driven by message-passing, where node representations are iteratively updated via local neighborhood aggregation. Despite their success, message-passing suffers from fundamental limitations -- including constrained expressiveness, over-smoothing, over-squashing, and limited capacity to model long-range dependencies. These issues hinder scalability: increasing data size or model size often fails to yield improved performance. To this end, we explore pathways beyond message-passing and introduce Generative Graph Pattern Machine (G$^2$PM), a generative Transformer pre-training framework for graphs. G$^2$PM represents graph instances (nodes, edges, or entire graphs) as sequences of substructures, and employs generative pre-training over the sequences to learn generalizable and transferable representations. Empirically, G$^2$PM demonstrates strong scalability: on the ogbn-arxiv benchmark, it continues to improve with model sizes up to 60M parameters, outperforming prior generative approaches that plateau at significantly smaller scales (e.g., 3M). In addition, we systematically analyze the model design space, highlighting key architectural choices that contribute to its scalability and generalization. Across diverse tasks -- including node/link/graph classification, transfer learning, and cross-graph pretraining -- G$^2$PM consistently outperforms strong baselines, establishing a compelling foundation for scalable graph learning. The code and dataset are available at https://github.com/Zehong-Wang/G2PM.

图神经网络生成模型可扩展性Transformer

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。