用邻居采样+动量改进图神经网络训练，收敛更快更稳定。
Neighbor-Sampling Based Momentum Stochastic Methods for Training Graph Neural Networks
- 基于邻居采样设计带动量的自适应优化方法
- 在多个数据集上显著优于传统采样SGD，大图表现尤佳
- 结合控制变量降低采样误差，理论收敛性有保证
图卷积网络（GCNs）是图表示学习的强大工具。由于其递归邻域聚合机制，现有高效训练方法缺乏理论保障，或缺少现代深度学习中的自适应与动量等关键要素。本文提出几种基于邻居采样（NS）的Adam型随机优化方法，用于求解非凸的GCN训练问题。我们采用[1]提出的控制变量技术，降低邻居采样带来的随机误差。在标准Adam类方法假设下，证明了所提方法具有最优收敛速率。此外，在多个基准数据集的节点分类任务上进行了大量实验，结果表明，相比使用控制变量的经典NS-SGD，本方法性能更优，尤其在大规模图数据上优势明显。代码已开源：https://github.com/RPI-OPT/CV-ADAM-GNN。
原文摘要 · Abstract (English)
Graph convolutional networks (GCNs) are a powerful tool for graph representation learning. Due to the recursive neighborhood aggregations employed by GCNs, efficient training methods suffer from a lack of theoretical guarantees or are missing important practical elements from modern deep learning algorithms, such as adaptivity and momentum. In this paper, we present several neighbor-sampling (NS) based Adam-type stochastic methods for solving a nonconvex GCN training problem. We utilize the control variate technique proposed by [1] to reduce the stochastic error caused by neighbor sampling. Under standard assumptions for Adam-type methods, we show that our methods enjoy the optimal convergence rate. In addition, we conduct extensive numerical experiments on node classification tasks with several benchmark datasets. The results demonstrate superior performance of our methods over classic NS-based SGD that also uses the control-variate technique, especially for large-scale graph datasets. Our code is available at https://github.com/RPI-OPT/CV-ADAM-GNN .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。