提出半精度训练的图神经网络系统，显著提速降存且保持精度。
Optimization of GNN Training Through Half-precision
- 设计新型半精度向量操作与量化SpMM，解决溢出与资源浪费问题。
- 在GAT/GCN/GIN上平均提速2.30倍，内存减少2.67倍，精度相近。
- 适合追求高效推理与低显存部署的图学习研究者与工程师。
近期低精度训练趋势（如半精度浮点）在深度学习中展现出提升系统性能与降低内存消耗的优势，同时保持精度。然而，现有图神经网络（GNN）系统在使用半精度时表现严重不足，出现异常精度下降与性能损失。分析表明，其根源在于精度降低引发的数值溢出、硬件资源利用率低及训练效率差。为此，本文提出HalfGNN，一种基于半精度的GNN系统。该系统引入两项核心技术：面向半精度数据类型的新型向量运算，提升数据加载与归约效率；以及离散化SpMM，有效缓解数值溢出并原生实现负载均衡。这些改进提升了硬件利用率，降低内存占用，并消除原子写操作。实验结果表明，HalfGNN在GAT、GCN和GIN模型上相比基于float的DGL系统，平均训练速度提升2.30倍，内存消耗减少2.67倍，同时保持相近的模型精度。
原文摘要 · Abstract (English)
Recent trends in lower precision, e.g. half-precision floating point, training have shown improved system performance and reduced memory usage for Deep Learning while maintaining accuracy. However, current GNN systems cannot achieve such goals for GNN, as our analyses show that they massively underperform while showing abnormal accuracy when using half-precision. These systems suffer from value overflow issues due to lowered precision, under-utilization of hardware resources, and poor training performance. To mitigate this, we introduce HalfGNN, a half-precision based GNN system. HalfGNN proposes novel techniques: new vector operations for half-precision data types that improve data load and reduction performance, and discretized SpMM that overcomes the value overflow and natively provides workload balancing. Such techniques improve hardware utilization, reduce memory usage, and remove atomic writes. Evaluations show that HalfGNN achieves on average of 2.30X speedup in training time over DGL (float-based) for GAT, GCN, and GIN respectively while achieving similar accuracy, and saving 2.67X memory.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。