用合成图预训练的图基础模型,提升小样本学习性能。
GraphPFN: A Prior-Data Fitted Graph Foundation Model
- 基于先验数据拟合框架,生成带属性的合成图用于预训练。
- 在多种真实图数据集上,小样本和微调表现均优于现有方法。
- 适合缺乏标注数据的图学习场景,如医疗或金融网络分析。
图基础模型面临跨领域迁移性和数据稀缺性等根本挑战,制约其可行性。然而,表格领域已出现成功的基础模型(如TabPFN),其基于先验-数据拟合网络(PFN)框架,在精心设计的合成数据上预训练,实现上下文学习。近期,G2T-FM将图节点任务转化为表格任务,但仅使用人工特征且未在图数据上预训练。本文提出GraphPFN,一种专为图节点任务设计的基于PFN的模型。我们通过多层级随机块模型与偏好连接过程生成结构,结合图感知的结构化因果模型生成属性,构建合成图先验分布。随后,将表格式基础模型LimiX扩展为支持注意力聚合的图邻域信息融合层,并在数百万张从该先验采样的合成图上进行训练。在多个真实世界图数据集的节点级任务中,GraphPFN在上下文学习和微调设置下均达到当前最优性能,显著超越G2T-FM、先前图基础模型及从头训练的专用GNN。结果表明,基于PFN的模型具备构建图基础模型的巨大潜力。
原文摘要 · Abstract (English)
Graph foundation models face several fundamental challenges including transferability across diverse domains and data scarcity, which calls into question the very feasibility of creating such models. However, despite similar challenges, the tabular domain has recently witnessed the emergence of the first successful foundation models such as TabPFN. These models are based on the prior-data fitted networks (PFN) framework, in which models are pretrained on carefully designed synthetic datasets to make predictions in an in-context learning setting. Recently, G2T-FM, a framework that converts graph node-level tasks into tabular tasks, has made the first step towards adopting PFNs for graphs, yet it is limited to hand-crafted features and was never pretrained on graph data. In this work, we make the next step by proposing GraphPFN, a PFN-based model designed and pretrained specifically for graph node-level tasks. Following the PFN framework, we first design a prior distribution of synthetic attributed graphs by using a novel combination of multi-level stochastic block models and a preferential attachment process for structure generation and graph-aware structured causal models for attribute generation. Then, we augment the tabular foundation model LimiX with attention-based graph neighborhood aggregation layers and train it on millions of synthetic graphs sampled from our prior. On diverse real-world graph datasets with node-level tasks, GraphPFN achieves state-of-the-art results in both in-context learning and finetuning regimes, outperforming G2T-FM, prior GFMs, and task-specific GNNs trained from scratch. More broadly, GraphPFN shows the potential of PFN-based models for building graph foundation models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。