用控制理论解释图神经网络梯度消失,提出无额外参数的新方法缓解过平滑和过挤压。
On Vanishing Gradients, Over-Smoothing, and Over-Squashing in GNNs: Bridging Recurrent and Graph Learning
- 将GNN视为循环系统,用状态空间建模统一分析梯度消失问题。
- 证明几层后梯度就极度消失,过平滑由梯度消失机制直接导致。
- 无需增加参数即可有效缓解过平滑与过挤压,适合深度图学习研究者。
图神经网络(GNN)通过消息传递机制利用图结构在节点间传递信息,但该方法易受过平滑和过挤压现象影响,导致层数增加时表征崩溃,以及对远距离或连接差的节点信息不敏感。本文从梯度消失角度,借助线性控制理论,提出将GNN视为递归模型的统一视角。我们实证表明,采用简单的状态空间形式化可有效缓解过平滑与过挤压,且无需额外训练参数。理论与实验共同揭示:(i) GNN设计上在数层后即存在极端梯度消失;(ii) 过平滑与梯度消失机制直接相关;(iii) 过挤压最易通过图重连与梯度消失缓解相结合来解决。本工作有望弥合递归模型与图神经网络研究间的鸿沟,推动新型深层高性能GNN的设计。
原文摘要 · Abstract (English)
Graph Neural Networks (GNNs) are models that leverage the graph structure to transmit information between nodes, typically through the message-passing operation. While widely successful, this approach is well known to suffer from the over-smoothing and over-squashing phenomena, which result in representational collapse as the number of layers increases and insensitivity to the information contained at distant and poorly connected nodes, respectively. In this paper, we present a unified view of these problems through the lens of vanishing gradients, using ideas from linear control theory for our analysis. We propose an interpretation of GNNs as recurrent models and empirically demonstrate that a simple state-space formulation of a GNN effectively alleviates over-smoothing and over-squashing at no extra trainable parameter cost. Further, we show theoretically and empirically that (i) GNNs are by design prone to extreme gradient vanishing even after a few layers; (ii) Over-smoothing is directly related to the mechanism causing vanishing gradients; (iii) Over-squashing is most easily alleviated by a combination of graph rewiring and vanishing gradient mitigation. We believe our work will help bridge the gap between the recurrent and graph neural network literature and will unlock the design of new deep and performant GNNs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。