MixGCN通过混合并行与加速器提升图神经网络训练效率
MixGCN: Scalable GCN Training by Mixture of Parallelism and Mixture of Accelerators
- 融合多种并行策略与稀疏/密集加速器协同
- 通信量恒定且负载均衡，支持大规模图训练
- 适合需要高效训练大图模型的研究者
图卷积网络（GCNs）在基于图的学习任务中表现优异，但全图训练面临两大挑战：一是特征张量易超出现代加速器的内存容量并阻塞通信带宽；二是训练过程中稀疏与密集矩阵运算交替，难以高效利用计算资源。现有分布式训练方案多采用分片并行，仅部分缓解第一问题且通信开销随规模增长。为此，我们提出MixGCN，同时应对两大挑战。为解决内存与通信瓶颈，引入混合并行策略，理论与实证分析均验证其通信量恒定且负载更均衡；针对计算模式切换问题，采用混合加速器架构，配备专用加速单元和细粒度流水线。大量实验表明，MixGCN显著提升训练效率与可扩展性。
原文摘要 · Abstract (English)
Graph convolutional networks (GCNs) have demonstrated superiority in graph-based learning tasks. However, training GCNs on full graphs is particularly challenging, due to the following two challenges: (1) the associated feature tensors can easily explode the memory and block the communication bandwidth of modern accelerators, and (2) the computation workflow in training GCNs alternates between sparse and dense matrix operations, complicating the efficient utilization of computational resources. Existing solutions for scalable distributed full-graph GCN training mostly adopt partition parallelism, which is unsatisfactory as they only partially address the first challenge while incurring scaled-out communication volume. To this end, we propose MixGCN aiming to simultaneously address both the aforementioned challenges towards GCN training. To tackle the first challenge, MixGCN integrates mixture of parallelism. Both theoretical and empirical analysis verify its constant communication volumes and enhanced balanced workload; For handling the second challenge, we consider mixture of accelerators (i.e., sparse and dense accelerators) with a dedicated accelerator for GCN training and a fine-grain pipeline. Extensive experiments show that MixGCN achieves boosted training efficiency and scalability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。