提出可预测Transformer微调能耗的模型，助力绿色AI系统设计
The Energy Consumption of Transformer Fine-Tuning: A Roofline-Inspired Scaling Model
- 基于屋顶线模型思想，用计算、内存流量和硬件效率代理能耗
- 在多GPU配置下准确预测训练能耗，误差小且覆盖异构环境
- 适合关注模型训练能效、绿色计算的研究者与工程师
基于Transformer的模型支撑现代自然语言处理，但其计算与能源成本快速上升。随着模型规模和并行度的扩大，精确预测能耗对可持续和成本敏感的系统设计至关重要。本文提出一个针对多GPU上Transformer训练能耗的建模框架。通过控制BERT模型的架构变化实验，将实测能耗与计算量、内存流量及硬件效率的轻量级代理变量关联。受屋顶线模型启发，引入基于加速比的硬件效率因子，以捕捉张量并行和全分片数据并行的影响。由此推导出的缩放定律模型，可在异构配置下准确预测训练能耗。
原文摘要 · Abstract (English)
Transformer-based models underpin modern natural language processing but incur rapidly growing computational and energy costs. As training scales in both model size and parallelism, accurately predicting energy consumption has become critical for sustainable and cost-aware system design. We present a framework for modeling the energy consumption of Transformer training on multiple GPUs. Using controlled architectural sweeps of BERT models, we relate measured energy to lightweight proxies for compute, memory traffic, and hardware efficiency. Inspired by roofline models, our approach incorporates a speedup-based hardware-efficiency factor that captures the effects of tensor parallelism and fully sharded data parallelism. We derive a scaling law model that accurately predicts training energy across heterogeneous configurations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。