研究并行训练中能耗与资源消耗的关系，揭示能耗可预测规律。
Energy Consumption in Parallel Neural Network Training
- 通过控制GPU数量、批量大小等参数，系统测试了并行训练的能耗
- 能耗与GPU小时数近似线性相关，但系数因模型和硬件而异
- 为可持续AI训练提供量化参考，适合关注能效的研究者
神经网络训练对计算资源的需求不断增长，导致能源消耗显著上升。尽管并行化使模型和数据规模得以扩展并加快训练速度，但其对能源消耗的影响常被忽视。为填补这一研究空白，我们对ResNet50和FourCastNet两种模型进行了数据并行训练的缩放实验，评估了GPU数量、全局批量大小和本地批量大小等并行化参数对预测性能、训练时间和能源消耗的影响。结果表明，能源消耗与所用资源（即GPU小时数）近似呈线性关系；然而，不同模型训练和硬件平台下的比例系数差异显著，并且系统性地受每GPU小时样本数和梯度更新次数的影响。这些发现揭示了大规模神经网络训练扩缩与能源消耗之间复杂的相互作用，可为未来更可持续的AI研究提供指导。
原文摘要 · Abstract (English)
The increasing demand for computational resources of training neural networks leads to a concerning growth in energy consumption. While parallelization has enabled upscaling model and dataset sizes and accelerated training, its impact on energy consumption is often overlooked. To close this research gap, we conducted scaling experiments for data-parallel training of two models, ResNet50 and FourCastNet, and evaluated the impact of parallelization parameters, i.e., GPU count, global batch size, and local batch size, on predictive performance, training time, and energy consumption. We show that energy consumption scales approximately linearly with the consumed resources, i.e., GPU hours; however, the respective scaling factor differs substantially between distinct model trainings and hardware, and is systematically influenced by the number of samples and gradient updates per GPU hour. Our results shed light on the complex interplay of scaling up neural network training and can inform future developments towards more sustainable AI research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。