解决图神经网络训练中历史特征过时问题，提升大规模图模型性能与收敛速度。
Haste Makes Waste: A Simple Approach for Scaling Graph Neural Networks
- 利用动态更新机制减少历史嵌入的过时性，降低计算偏差。
- 在ogbn-papers100M和ogbn-products上分别提升2.7%和3.6%性能。
- 方法简单易集成，适合大规模图神经网络训练场景。
图神经网络（GNN）在图表示学习中表现优异，已有多种采样方法用于扩展其在大规模图上的应用。一些有前景的训练算法通过利用历史嵌入来降低计算与内存开销，同时保持模型表达能力。然而，这些方法因特征历史过时而引入显著计算偏差，在大规模问题上表现不佳。本文系统分析了该类方法的过时现象及其性能下降原因。基于发现，提出一种简单但高效的训练算法REST，有效缓解特征过时问题，显著提升不同批量大小下的性能与收敛速度。该方法可无缝集成至现有方案，实现轻松部署。大量实验表明，对现有历史嵌入方法的改进，在ogbn-papers100M和ogbn-products数据集上分别带来2.7%和3.6%的性能提升，并显著加速收敛。
原文摘要 · Abstract (English)
Graph neural networks (GNNs) have demonstrated remarkable success in graph representation learning, and various sampling approaches have been proposed to scale GNNs to applications with large-scale graphs. A class of promising GNN training algorithms take advantage of historical embeddings to reduce the computation and memory cost while maintaining the model expressiveness of GNNs. However, they incur significant computation bias due to the stale feature history. In this paper, we provide a comprehensive analysis of their staleness and inferior performance on large-scale problems. Motivated by our discoveries, we propose a simple yet highly effective training algorithm (REST) to effectively reduce feature staleness, which leads to significantly improved performance and convergence across varying batch sizes. The proposed algorithm seamlessly integrates with existing solutions, boasting easy implementation, while comprehensive experiments underscore its superior performance and efficiency on large-scale benchmarks. Specifically, our improvements to state-of-the-art historical embedding methods result in a 2.7% and 3.6% performance enhancement on the ogbn-papers100M and ogbn-products dataset respectively, accompanied by notably accelerated convergence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。