随机采点训练GNN效果竟优于全图,还更省时省内存。
Implicit Regularization of Mini-Batch Training in Graph Neural Networks
- 用随机节点采样构建子图,无需保持局部结构。
- 采样后损失更接近全图损失,梯度方差更低。
- 适用于大规模图神经网络快速训练,理论依据强。
图神经网络的分批训练与独立同分布数据训练本质不同:采样子图会改变拓扑并引入边界效应,先前工作为此设计了保持局部连通性的结构感知采样器以降低嵌入方差。令人惊讶的是,我们发现最简单的随机节点采样(RNS)——在均匀采样的节点诱导的子图上训练——在10个数据集中的8个上表现匹配或超越全图训练,且耗时和内存仅为后者的几分之一。通过反向误差分析,我们揭示图分批随机梯度下降(SGD)隐式最小化采样损失加上一个与批次梯度方差成正比的正则项,该量直接受采样器影响。尽管RNS丢弃局部结构,但其生成的子图期望损失更接近全图损失,且每批次梯度方差更低,从而形成更优的隐式目标。本分析将采样器选择重新理解为一种隐式正则化,确认RNS是一种强大且理论扎实的可扩展GNN训练方法。
原文摘要 · Abstract (English)
Mini-batch training of Graph Neural Networks (GNNs) is fundamentally different from training on i.i.d. data: sampling a subgraph alters the topology and introduces boundary effects, leading prior work to develop structure-aware samplers that preserve local connectivity and reduce embedding variance. Surprisingly, we demonstrate that the simplest possible scheme, Random Node Sampling (RNS), training on the induced subgraph of uniformly sampled nodes, matches or outperforms full-graph training on 8 of 10 datasets at a fraction of the wall-clock time and memory. To explain this, we apply backward error analysis to graph mini-batch Stochastic Gradient Descent (SGD) and show that it implicitly minimizes the sampled loss plus a regularizer proportional to the mini-batch gradient variance, a quantity directly shaped by the sampler. Although RNS discards local structure, it produces mini-batches whose expected loss is closer to the full-graph loss, and whose per-batch gradients have lower variance, yielding a better implicit objective. Our analysis reframes the choice of graph sampler as a form of implicit regularization, and identifies RNS as a strong, theoretically grounded method for scalable GNN training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。