揭示LLM推理能耗与计费方式的差异，提出联合优化请求与单令牌能耗的新思路。
Characterization of Request and Token Energy Costs for LLM Inference Workloads on GPU Platforms
- 分解能耗为预填充固定成本和每输出令牌的边际成本
- 输出长度增加时单令牌能耗下降，但总能耗上升，批处理增益受上下文长度限制
- 混合专家模型通过批处理显著缩小与稠密模型的能耗差距
大语言模型（LLM）推理服务按令牌计价，但GPU能耗发生在推理窗口期内。这种计费与能耗不匹配导致以令牌归一化的指标不完整：即使总请求能耗上升，平均输出令牌能耗仍可能下降。本文构建了分解式能耗模型——固定预填充阶段与生成设置开销，每生成一个输出令牌增加边际能耗。在NVIDIA H100和H200 GPU上对稠密与混合专家（MoE）模型进行评估，报告了请求能耗与令牌能耗随模型类型（M）、阶段（P）、批大小（B）、上下文长度（C）和输出长度（N）的变化。以Llama-3.2-1B在H200上为例，批大小16、上下文4K时，输出长度从10增至512，单令牌能耗从7.46降至0.72焦耳/令牌，而批量推理窗口总能耗从1.19升至5.93千焦。批处理可降低单令牌能耗，但收益受上下文长度约束：输出10令牌时，批大小16到1的能耗降幅从512上下文的6.31倍降至4K上下文的1.17倍。MoE模型加剧此效应：稀疏路由与专家碎片化执行增加了低并发下的固定能耗，而批处理将该能耗分摊至更多输出令牌，显著缩小稠密模型与MoE模型间的单令牌能耗差距。结果表明，能耗感知的服务应同时优化请求能耗与单令牌能耗，而非仅追求降低单位令牌成本。
原文摘要 · Abstract (English)
Large language model (LLM) inference serving is priced by tokens, but GPU energy is consumed over inference windows. This accounting mismatch makes token-normalized metrics incomplete, since average output-token energy can decrease even when total request energy increases. We characterize this behavior with a decomposed energy model: a fixed one-time prefill with a fixed generation setup cost, while each output-token generation step adds marginal step energy. We evaluate this LLM inference energy model on NVIDIA H100 and H200 GPUs across dense and mixture-of-experts (MoE) models, reporting both request energy and token energy as functions of model type (M), phase (P), batch size (B), context length (C), and output length (N). For Llama-3.2-1B on H200 at batch-16 and context-4K, increasing output length from 10 to 512 tokens reduces token energy from 7.46 to 0.72 J/token while total batched inference-window energy increases from 1.19 to 5.93 kJ. Batching also reduces token energy, but the gain is context-bounded: at 10 output tokens, the batch-16 to batch-1 gain falls from 6.31x at context-512 to 1.17x at context-4K. MoE models amplify this effect: sparse routing and fragmented expert execution increase fixed energy at low concurrency, while batching spreads that energy across more generated tokens and substantially narrows the dense-vs.-MoE token-energy gap. These results show that energy-aware serving should jointly optimize both request energy and token energy, rather than only reducing per-token energy cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。