用可解释的能效权衡界面优化模型性能与能耗,助力绿色AI。
Optimising for Energy Efficiency and Performance in Machine Learning
- 提出能量优化调参器ECOpt,量化性能与能效的权衡关系。
- 发现参数量和浮点运算数不能可靠反映真实能耗,且不同硬件上变压器模型能效相近。
- 帮助发现7个兼具高精度与低能耗的CIFAR-10模型,适合关注碳足迹的研究者。
机器学习(ML)的普及和模型规模持续扩大,导致能源消耗和环境影响加剧。然而,关于机器学习中能源消耗的规律尚不明确,现有研究多聚焦训练成本,忽视了更大的推理能耗。此外,现有测量工具无法提供可操作的反馈。为填补这些空白,我们开发了能量消耗优化器(ECOpt):一种同时优化能效与模型性能的超参数调优工具。ECOpt将这两项指标之间的权衡以可解释的帕累托前沿形式呈现,使机器学习从业者能在最大化模型效益的同时,合理评估能源成本与环境影响,并符合新法规要求。利用ECOpt,我们发现参数量和浮点运算次数不能作为能耗的可靠代理指标,并观察到文本生成用的Transformer模型在不同硬件上的能效相对一致。这些发现推动了对机器学习模型能源指标的测量与公开。我们进一步证明,使用ECOpt可带来净环境效益,并从中发现了7个在准确率与能效综合表现上优于当前最佳水平的CIFAR-10模型。
原文摘要 · Abstract (English)
The ubiquity of machine learning (ML) and the demand for ever-larger models bring an increase in energy consumption and environmental impact. However, little is known about the energy scaling laws in ML, and existing research focuses on training cost -- ignoring the larger cost of inference. Furthermore, tools for measuring the energy consumption of ML do not provide actionable feedback. To address these gaps, we developed Energy Consumption Optimiser (ECOpt): a hyperparameter tuner that optimises for energy efficiency and model performance. ECOpt quantifies the trade-off between these metrics as an interpretable Pareto frontier. This enables ML practitioners to make informed decisions about energy cost and environmental impact, while maximising the benefit of their models and complying with new regulations. Using ECOpt, we show that parameter and floating-point operation counts can be unreliable proxies for energy consumption, and observe that the energy efficiency of Transformer models for text generation is relatively consistent across hardware. These findings motivate measuring and publishing the energy metrics of ML models. We further show that ECOpt can have a net positive environmental impact and use it to uncover seven models for CIFAR-10 that improve upon the state of the art, when considering accuracy and energy efficiency together.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。