提出新方法解决图神经网络训练中的梯度过平滑与爆炸问题。
Taming Gradient Oversmoothing and Expansion in Graph Neural Networks
- 发现深层图神经网络存在梯度过平滑和残差连接引发的梯度膨胀。
- 通过约束每层的Lipschitz常数，实现对梯度膨胀的有效抑制。
- 所提归一化方法可支持数百层残差GNN高效训练，适合深度图模型研究者。
多层图神经网络（GNN）常受过平滑问题困扰。现有研究虽分析了过平滑现象，但未关注其在优化过程中的影响。本文揭示了训练中存在梯度过平滑现象，阻碍优化进程；同时发现，虽残差连接被广泛用于缓解梯度消失，却会引入梯度膨胀——即梯度在不同方向上剧烈发散。因此，单纯依赖残差结构无法真正解决深度模型训练难题。我们理论分析表明，通过约束各层的Lipschitz常数可中和梯度膨胀。基于此，提出一种简单有效的归一化方法以防止梯度膨胀。实验证明，结合该方法的残差GNN可在不牺牲性能的前提下实现数百层的高效训练，且结果与理论预测一致。
原文摘要 · Abstract (English)
Oversmoothing has been claimed as a primary bottleneck for multi-layered graph neural networks (GNNs). Multiple analyses have examined how and why oversmoothing occurs. However, none of the prior work addressed how optimization is performed under the oversmoothing regime. In this work, we show the presence of $\textit{gradient oversmoothing}$ preventing optimization during training. We further analyze that GNNs with residual connections, a well-known solution to help gradient flow in deep architecture, introduce $\textit{gradient expansion}$, a phenomenon of the gradient explosion in diverse directions. Therefore, adding residual connections cannot be a solution for making a GNN deep. Our analysis reveals that constraining the Lipschitz bound of each layer can neutralize the gradient expansion. To this end, we provide a simple yet effective normalization method to prevent the gradient expansion. An empirical study shows that the residual GNNs with hundreds of layers can be efficiently trained with the proposed normalization without compromising performance. Additional studies show that the empirical observations corroborate our theoretical analysis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。