提出细粒度能耗分析方法，揭示Transformer组件真实能效差异。
Dissecting Transformers: A CLEAR Perspective towards Green AI
- 通过重复执行捕捉微秒级组件能耗，突破毫秒级传感器限制。
- 发现注意力模块每计算浮点操作耗能远高于模型整体，FLOPs不反映真实能耗。
- 适用于关注绿色AI、模型优化与能效建模的研究者。
大型语言模型的快速普及引发了严重的环境担忧。与一次性训练成本不同，模型推理持续进行，成为人工智能能耗的主要来源。然而，现有可持续性研究多仅报告粗粒度的模型级指标，将能效视为次要考量。为此，本文提出组件级能耗评估方法CLEAR（Component-Level Energy Assessment via Repetitions），克服微秒级组件执行与毫秒级能量传感器监测之间的时序错配问题。利用CLEAR，我们对15个涵盖四种架构类型的模型进行了评估，保持组件间能耗方差低于9.5%，并捕捉超过90%的总能耗作为独立组件。首次实现对变压器组件在批大小、注意力头数、隐藏维度、键值缓存及注意力变体等关键参数下的全面细粒度能耗分析。结果表明，注意力模块单位FLOP耗能显著高于整个模型，说明仅用FLOPs无法准确反映组件级真实能耗。CLEAR实现了可靠的细粒度能耗测量，并为能耗预测建模提供了坚实的理论基础。
原文摘要 · Abstract (English)
The rapid adoption of Large Language Models (LLMs) has raised significant environmental concerns. Unlike the one-time cost of training, LLM inference occurs continuously and dominates the AI energy footprint. Yet most sustainability studies report only coarse model-level metrics, treating energy efficiency as an afterthought rather than a primary objective. Addressing the limitation, we propose Component-Level Energy Assessment via Repetitions CLEAR, to overcome temporal mismatch between microsecond scale component execution and millisecond(ms) scale monitoring of energy sensors. Using CLEAR, we evaluate 15 models spanning four architecture types, keeping component-wise energy variance below 9.5% while capturing over 90% of total energy as individual components. We present the first comprehensive, fine-grained energy analysis of Transformer components across key parameters such as batch size, attention heads, hidden dimension, KV cache, and attention variants. Our findings reveal that Attention consumes significantly more Energy per FLOP as compared to the entire model, indicating that FLOPs alone fail to capture true component-level energy cost. CLEAR enables reliable fine-grained energy measurements and provides a strong formal foundation for predictive modelling of energy consumption.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。