GraphGen+实现分布式图学习的高效子图生成，无需预存数据。
GraphGen+: Advancing Distributed Subgraph Generation and Graph Learning On Industrial Graphs
- 将分布式子图生成与内存内学习同步，无需外部存储。
- 子图生成速度比传统方法快27倍，训练支持每轮100万节点。
- 适合工业级大规模图学习，解决存储与性能双重瓶颈。
基于图的计算在众多应用中至关重要，图规模可达到万亿级边。为实现此类大图的高效训练，通常采用小批量子图采样，避免将整个图加载到内存中。然而现有方案存在显著权衡：在线子图生成（如DGL、PyG）受限于单机，导致严重性能瓶颈；离线预计算子图（如GraphGen）虽提升采样效率，却带来巨大存储开销和训练时的高I/O成本。为此，我们提出 extbf{GraphGen+}，一个集成框架，实现分布式子图生成与内存内图学习的同步，无需外部存储即可显著提升效率。GraphGen+相比传统SQL-like方法实现27×的子图生成加速，较GraphGen提升1.3×，支持每迭代100万节点的训练，消除预计算子图带来的开销，是工业级图学习的可扩展且实用的解决方案。
原文摘要 · Abstract (English)
Graph-based computations are crucial in a wide range of applications, where graphs can scale to trillions of edges. To enable efficient training on such large graphs, mini-batch subgraph sampling is commonly used, which allows training without loading the entire graph into memory. However, existing solutions face significant trade-offs: online subgraph generation, as seen in frameworks like DGL and PyG, is limited to a single machine, resulting in severe performance bottlenecks, while offline precomputed subgraphs, as in GraphGen, improve sampling efficiency but introduce large storage overhead and high I/O costs during training. To address these challenges, we propose \textbf{GraphGen+}, an integrated framework that synchronizes distributed subgraph generation with in-memory graph learning, eliminating the need for external storage while significantly improving efficiency. GraphGen+ achieves a \textbf{27$\times$} speedup in subgraph generation compared to conventional SQL-like methods and a \textbf{1.3$\times$} speedup over GraphGen, supporting training on 1 million nodes per iteration and removing the overhead associated with precomputed subgraphs, making it a scalable and practical solution for industry-scale graph learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。