提出自适应分布式训练框架，让图Transformer在大规模图上高效训练。
Scalable and Adaptive Parallel Training of Graph Transformer on Large Graphs
- 根据图结构和硬件自动选择最优并行策略
- 稀疏注意力加速3.8倍，内存降低78%
- 8卡下提速6倍，适合大图预训练场景
图基础模型通过在大规模图上进行预训练，展现出对多种下游任务的强大适应性。然而，现有图Transformer实现通常受限于单卡系统，导致在大规模图上训练时间过长或显存溢出。此外，将图Transformer训练完全并行化面临挑战，其效率高度依赖图结构与系统特性（如带宽、内存容量）。本文提出一种分布式训练框架，能根据图结构和硬件配置自动选择并优化并行策略。通过实现分布式稀疏操作，本框架将稀疏图注意力计算加速达3.8倍，内存消耗降低78%。在多个大规模图基准测试中，该框架在8张GPU下实现最高6倍的加速。结果表明，所提框架显著提升了图Transformer的可扩展性，使其更接近实用化图基础模型的目标。
原文摘要 · Abstract (English)
Graph foundation models have demonstrated remarkable adaptability across diverse downstream tasks through large-scale pretraining on graphs. However, existing implementations of the backbone model, graph transformers, are typically limited to single-GPU systems, leading to long training times or out-of-memory issues on large graphs. Moreover, parallelizing graph transformer training over the full graph is challenging, as efficiency depends heavily on both the graph structure and system characteristics, such as bandwidth and memory capacity. In this work, we introduce a distributed training framework for graph transformers, which automatically selects and optimizes parallelization strategies based on the graph structure and hardware configuration. With our implementation of distributed sparse operations, we accelerate sparse graph attention by up to 3.8x and reduce memory consumption by 78% compared to state-of-the-art frameworks. On large graph benchmarks, our proposed framework achieves up to 6x speedup with system scaling up to 8 GPUs. These results demonstrate that the proposed framework improves the scalability of graph transformers, bringing them closer to serving as practical graph foundation models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。