arXiv:2608.25096cs.LGcs.AI2026-08

对比不同注意力架构的推理能耗,发现滑动窗口能大幅降低长文本生成耗能。

Understanding the Energy Scaling of Large Language Model Inference Across Context Lengths and Attention Architectures

论文配图:Understanding the Energy Scaling of Large Language Model Inference Across Context Lengths and Attention Architectures
图 1 · 摘自论文原文
  • 测试MHA、GQA及带滑窗的GQA在不同上下文长度下的能耗表现。
  • 使用滑窗的GQA模型能耗几乎不随上下文变长而上升。
  • 批量处理可降低87%每词能耗,适合大规模部署优化。

大语言模型(LLM)推理的能源消耗和环境影响日益受到关注。本文对采用多头注意力(MHA)、分组查询注意力(GQA)及带滑动窗口注意力(SWA)的代表性开源模型,在不同上下文长度、批大小和生成负载下,进行了系统性的解码阶段能耗实证研究。通过NVIDIA硬件计数器测量GPU能耗,分析了上下文长度、注意力机制、键值(KV)缓存增长和批处理对解码能耗的影响。结果表明,注意力机制是决定解码能耗随上下文长度变化的主要因素:MHA模型能耗增长显著快于GQA模型;而采用滑窗的GQA模型则几乎保持恒定能耗。此外,模型规模主要决定绝对能耗,批量处理可将每生成词的能耗和请求延迟降低高达87%。这些发现为选择节能型模型架构与推理配置提供了实用指导。

原文摘要 · Abstract (English)

The growing adoption of large language models (LLMs) has raised increasing concerns about the energy consumption and environmental impact of inference. This paper presents a systematic empirical study of decode-phase energy consumption across representative open-source LLMs employing Multi-Head Attention (MHA), Grouped Query Attention (GQA), and Grouped Query Attention with Sliding Window Attention (SWA) to characterize how attention architecture influences decode-phase energy consumption under varying inference workloads. We evaluate four models across different context lengths, batch sizes, and generation workloads while measuring GPU energy using NVIDIA hardware counters. We examine the effects of context length, attention mechanism, Key-Value (KV) cache growth, and batching on decode-phase energy consumption. Results show that attention mechanism is the primary factor governing how decode energy scales with context length. MHA models exhibit substantially steeper energy growth than GQA models, whereas GQA with SWA maintains nearly constant energy consumption. We further show that model size primarily determines absolute energy consumption, while batching reduces both energy per generated token and request latency by up to 87%. These findings provide practical guidance for selecting energy-efficient LLM architectures and inference configurations.

LLM推理能耗分析注意力机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。