对比8种优化器，发现AdamW和NAdam最省电且高效。
An Analysis of Optimizer Choice on Energy Efficiency and Performance in Neural Network Training
- 在苹果M1 Pro上测了360次实验，比较不同优化器的能耗与性能。
- AdamW和NAdam能效比最优，SGD虽准确但耗电高。
- 适合关注绿色AI和训练成本的开发者参考。
随着机器学习模型日益复杂和计算需求增加，理解训练决策的环境影响对可持续人工智能发展至关重要。本文通过360次受控实验，系统研究了优化器选择与神经网络训练能效之间的关系。实验覆盖三个基准数据集（MNIST、CIFAR-10、CIFAR-100），采用八种主流优化器（SGD、Adam、AdamW、RMSprop、Adagrad、Adadelta、Adamax、NAdam），每种配置运行15个随机种子。基于CodeCarbon在Apple M1 Pro硬件上的精确能源追踪，我们测量了训练时长、峰值内存使用、二氧化碳排放量及最终模型性能。结果表明，训练速度、准确率与环境影响之间存在显著权衡，且因数据集和模型复杂度而异。在所有优化器中，AdamW和NAdam表现稳定且能效最高；而SGD在复杂数据集上虽精度更高，但碳排放也更显著。这些发现为追求性能与可持续性平衡的实践者提供了可操作的指导。
原文摘要 · Abstract (English)
As machine learning models grow increasingly complex and computationally demanding, understanding the environmental impact of training decisions becomes critical for sustainable AI development. This paper presents a comprehensive empirical study investigating the relationship between optimizer choice and energy efficiency in neural network training. We conducted 360 controlled experiments across three benchmark datasets (MNIST, CIFAR-10, CIFAR-100) using eight popular optimizers (SGD, Adam, AdamW, RMSprop, Adagrad, Adadelta, Adamax, NAdam) with 15 random seeds each. Using CodeCarbon for precise energy tracking on Apple M1 Pro hardware, we measured training duration, peak memory usage, carbon dioxide emissions, and final model performance. Our findings reveal substantial trade-offs between training speed, accuracy, and environmental impact that vary across datasets and model complexity. We identify AdamW and NAdam as consistently efficient choices, while SGD demonstrates superior performance on complex datasets despite higher emissions. These results provide actionable insights for practitioners seeking to balance performance and sustainability in machine learning workflows.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。