提出改进图注意力的ParaFormer，缓解深度图模型过平滑问题。
ParaFormer: A Generalized PageRank Graph Transformer for Graph Representation Learning
- 用PageRank增强注意力，实现自适应滤波
- 在11个数据集上提升节点与图分类性能
- 适合需要稳定全局信息捕捉的图学习任务
图变压器（GTs）凭借全连接特性，能有效捕获全局信息，成为有前景的图学习工具。为解决深度图神经网络（GNNs）的过平滑问题，早期引入全局注意力机制，但经实证与理论分析发现，该机制本身存在严重过平滑，导致节点表示不可区分，其低通滤波效应甚至强于传统GNNs。为此，我们提出PageRank Transformer（ParaFormer），其注意力模块融合PageRank机制，模拟深度变压器行为。理论与实验均表明，ParaFormer通过自适应滤波缓解过平滑。在包含数千至数百万节点的11个数据集上，其在节点分类与图分类任务中均表现一致提升，验证了有效性。补充材料（含代码与附录）见 https://github.com/chaohaoyuan/ParaFormer。
原文摘要 · Abstract (English)
Graph Transformers (GTs) have emerged as a promising graph learning tool, leveraging their all-pair connected property to effectively capture global information. To address the over-smoothing problem in deep GNNs, global attention was initially introduced, eliminating the necessity for using deep GNNs. However, through empirical and theoretical analysis, we verify that the introduced global attention exhibits severe over-smoothing, causing node representations to become indistinguishable due to its inherent low-pass filtering. This effect is even stronger than that observed in GNNs. To mitigate this, we propose PageRank Transformer (ParaFormer), which features a PageRank-enhanced attention module designed to mimic the behavior of deep Transformers. We theoretically and empirically demonstrate that ParaFormer mitigates over-smoothing by functioning as an adaptive-pass filter. Experiments show that ParaFormer achieves consistent performance improvements across both node classification and graph classification tasks on 11 datasets ranging from thousands to millions of nodes, validating its efficacy. The supplementary material, including code and appendix, can be found in https://github.com/chaohaoyuan/ParaFormer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。