揭示图神经网络过平滑的根源,提出可量化缓解的新方法。
Analyzing the Effect of Embedding Norms and Singular Values to Oversmoothing in Graph Neural Networks
- 用新指标MASED量化节点嵌入距离变化,推导层间上下界。
- 发现权重矩阵数量与邻接矩阵深度越多,过平滑越严重。
- 提出G-Reg正则化方案,深层模型性能超越浅层,适合冷启动场景。
本文研究深度图神经网络中过平滑现象的影响因素。基于新提出的度量指标均方距离均值(MASED),推导了层间MASED的上下界,并据此分析节点嵌入范数及权重矩阵奇异值对过平滑的影响。理论分析表明,可训练权重矩阵数量和邻接矩阵数量越多,过平滑越显著。基于此,提出将跳跃次数(邻接深度)与权重层数解耦的方法,引入G-Reg正则化方案以扩大边界,在多个数据集上验证其有效性:在深层网络中提升节点分类准确率,实现深度鲁棒性。特别地,在无标签节点特征的“冷启动”场景下,深层模型表现优于浅层。此外,通过MASED边界揭示感受野大小与性能间的权衡关系:将邻接跳数分布到少量可训练层中,避免参数过度或不足。
原文摘要 · Abstract (English)
In this paper, we study the factors that contribute to the effect of oversmoothing in deep Graph Neural Networks (GNNs). Specifically, our analysis is based on a new metric (Mean Average Squared Distance - $MASED$) to quantify the extent of oversmoothing. We derive layer-wise bounds on $MASED$, which aggregate to yield global upper and lower distance bounds. Based on this quantification of oversmoothing, we further analyze the importance of two different properties of the model; namely the norms of the generated node embeddings, along with the largest and smallest singular values of the weight matrices. Building on the insights drawn from the theoretical analysis, we show that oversmoothing increases as the number of trainable weight matrices and the number of adjacency matrices increases. We also use the derived layer-wise bounds on $MASED$ to form a proposal for decoupling the number of hops (i.e., adjacency depth) from the number of weight matrices. In particular, we introduce G-Reg, a regularization scheme that increases the bounds, and demonstrate through extensive experiments that by doing so node classification accuracy increases, achieving robustness at large depths. We further show that by reducing oversmoothing in deep networks, we can achieve better results in some tasks than using shallow ones. Specifically, we experiment with a ``cold start" scenario, i.e., when there is no feature information for the unlabeled nodes. Finally, we show empirically the trade-off between receptive field size (i.e., number of weight matrices) and performance, using the $MASED$ bounds. This is achieved by distributing adjacency hops across a small number of trainable layers, avoiding the extremes of under- or over-parameterization of the GNN.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。