用随机游走构建图的预训练模型，让Transformer学会理解不同结构的图数据。
Toward a Graph Foundation Model: Pre-Training Transformers With Random Walks
- 将节点表示为多条随机游走序列，使Transformer能处理异构图数据
- 设计新损失函数，在多种图上实现良好表征学习效果
- 适合需要通用图推理能力的研究者和工程应用
类似GPT等自然语言基础模型通过大规模数据预训练与Transformer架构展现出涌现能力，能否为图数据构建类似的基础模型？本文提出一种基于Transformer的图基础模型预训练方法。核心挑战在于如何让序列模型处理不同大小和领域图数据。为此，我们采用多个随机游走表示节点，使Transformer能从中提取节点、边及图的表示。提出一种新颖的上下文预测损失函数，并从理论上分析其区分邻域和图的能力。实验验证了该模型在下游任务中的可迁移性，展示了其作为图结构数据处理与推理基础的潜力。
原文摘要 · Abstract (English)
A foundation model like GPT elicits many emergent abilities, owing to the pre-training with broad inclusion of data and the use of the powerful Transformer architecture. While foundation models in natural languages are prevalent, can we build similar models for graphs? This paper describes an approach toward a graph foundation model that is pre-trained with diverse graph datasets by adapting the Transformer backbone. A central challenge toward this end is how a sequence model encodes graphs of varying sizes and from different domains. We propose representing a node as multiple random walks, such that the Transformer can extract node representations from sequences, which in turn form edge and graph representations. We develop a novel context prediction loss for these random walks and theoretically analyze their expressive power in distinguishing neighborhoods and graphs. We also demonstrate the pre-training of our model and its adaptation to downstream tasks, showcasing its potential as a foundation for processing and reasoning with graph-structured data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。