混合精度训练配合调参可降低回归模型能耗7-11瓦
Improve Machine Learning carbon footprint using Parquet dataset format and Mixed Precision training for regression models -- Part II
- 用Parquet格式+混合精度训练减少能耗
- 优化超参数后能耗降低7至11瓦
- 大规模集群实验可能改变统计结论
本研究为硕士论文第二部分，对比了使用逗号分隔值（CSV）与Parquet数据格式，在默认32位浮点和Nvidia混合精度（16位与32位）下训练回归模型的功耗。在与第一部分相同的定制PC上进行实验，设置不同深度神经网络（DNN）超参数（如批大小、神经元数、训练轮数）。以默认超参数的基准测试为参照，采用多种组合进行实验。结果记录于Excel，通过描述性统计计算组间均值，并以图表比较。结果显示，结合特定超参数的混合精度训练使功耗降低7至11瓦。但进一步使用ANOVA和T检验分析表明，回归测试中各组均值差异无统计显著性（接受原假设）。因此，当前实验规模下，模型技术与数据格式的选择对计算功耗及整体碳足迹无显著影响。然而，若采用多GPU集群扩大样本量，可能改变统计结果。
原文摘要 · Abstract (English)
This is the 2nd part of the dissertation for my master degree and compared the power consumption using the Comma-Separated-Values (CSV) and parquet dataset format with the default floating point (32bit) and Nvidia mixed precision (16bit and 32bit) while training a regression ML model. The same custom PC as per the 1st part, which was dedicated to the classification testing and analysis, was built to perform the experiments, and different ML hyper-parameters, such as batch size, neurons, and epochs, were chosen to build Deep Neural Networks (DNN). A benchmarking test with default hyper-parameter values for the DNN was used as a reference, while the experiments used a combination of different settings. The results were recorded in Excel, and descriptive statistics were chosen to calculate the mean between the groups and compare them using graphs and tables. The outcome was positive when using mixed precision combined with specific hyper-parameters. Compared to the benchmarking, optimising the regression models reduced the power consumption between 7 and 11 Watts. The regression results show that while mixed precision can help improve power consumption, we must carefully consider the hyper-parameters. A high number of batch sizes and neurons will negatively affect power consumption. However, this research required inferential statistics, specifically ANOVA and T-test, to compare the relationship between the means. The results reported no statistical significance between the means in the regression tests and accepted H0. Therefore, choosing different ML techniques and the Parquet dataset format will not improve the computational power consumption and the overall ML carbon footprint. However, a more extensive implementation with a cluster of GPUs can increase the sample size significantly, as it is an essential factor and can change the outcome of the statistical analysis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。