通过多尺度结构预训练,让图模型同时捕捉全局与局部特征。
Range-aware Positional Encoding via High-order Pretraining: Theory and Practice
- 用高阶自编码器重建节点的多尺度小波信号,学习图结构
- 理论证明可任意低误差预测邻接矩阵,具备强表达能力
- 不依赖领域数据,适合构建通用图结构编码器
在标签数据稀缺的实际应用中,如分子性质预测或材料科学,对海量图数据进行无监督预训练至关重要。现有方法针对特定图领域进行预训练,忽视了网络内部的固有联系,限制了知识迁移能力。本文提出一种新型图预训练策略,聚焦于建模图的多分辨率结构信息,可在保留节点周围局部结构的同时捕捉整个图的全局信息。我们扩展了Wavelet Positional Encoding(WavePE)的工作,通过预训练高阶置换等变自编码器(HOPE-WavePE),从多分辨率小波信号中重构节点连通性。与现有位置编码不同,本方法在下游任务中对图规模敏感,能高效捕捉图的全局结构。由于仅依赖图结构,该方法具有领域无关性,可适配多个领域的数据集,为构建通用图结构编码器和图基础模型铺平道路。我们理论上证明,此类架构存在参数化方式,可将输出邻接矩阵预测误差降至任意低水平。我们在不同领域的图级别预测任务上评估了HOPE-WavePE,结果表明其优于其他方法。
原文摘要 · Abstract (English)
Unsupervised pre-training on vast amounts of graph data is critical in real-world applications wherein labeled data is limited, such as molecule properties prediction or materials science. Existing approaches pre-train models for specific graph domains, neglecting the inherent connections within networks. This limits their ability to transfer knowledge to various supervised tasks. In this work, we propose a novel pre-training strategy on graphs that focuses on modeling their multi-resolution structural information, allowing us to capture global information of the whole graph while preserving local structures around its nodes. We extend the work of Wave}let Positional Encoding (WavePE) from (Ngo et al., 2023) by pretraining a High-Order Permutation-Equivariant Autoencoder (HOPE-WavePE) to reconstruct node connectivities from their multi-resolution wavelet signals. Unlike existing positional encodings, our method is designed to become sensitivity to the input graph size in downstream tasks, which efficiently capture global structure on graphs. Since our approach relies solely on the graph structure, it is also domain-agnostic and adaptable to datasets from various domains, therefore paving the wave for developing general graph structure encoders and graph foundation models. We theoretically demonstrate that there exists a parametrization of such architecture that it can predict the output adjacency up to arbitrarily low error. We also evaluate HOPE-WavePE on graph-level prediction tasks of different areas and show its superiority compared to other methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。