无需实测即可预测任意大模型在新显卡上的功耗与延迟。
WattGPU: Predicting Inference Power and Latency on Unseen GPUs and LLMs
- 仅用模型和显卡公开参数，构建功耗与延迟预测模型。
- 在未见过的显卡上功耗误差低于3.4%，延迟误差低于8.5%。
- 适合部署优化、硬件选型和能效评估的工程师与研究者。
大语言模型（LLM）推理负载正迅速成为数据中心能耗的主要来源。优化部署需将特定模型匹配至最高效的显卡，但当前缺乏无需全面测试即可完成匹配的工具。尽管已有部分预测模型，仍需实测数据且难以泛化到训练中未见的硬件。为此，我们提出WattGPU，包含两个预测模型：平均显卡功耗和令牌间延迟（ITL）。该方法仅依赖公开的LLM元数据与显卡规格，无需硬件访问或实测，可泛化至未见过的NVIDIA服务器级显卡及LLM。我们在涵盖42个开源LLM（0.1B–27B参数）与8款显卡的数据集上，采用留一显卡与留一模型交叉验证，在离线与服务器场景下进行评估。功耗模型在未见显卡上的中位绝对百分比误差分别为离线≤3.4%、服务器≤13.5%；延迟模型在服务器模式下误差≤8.5%，且保持较强的显卡排序相关性（Kendall τ≥0.76）。相比传统基于物理的基线（功耗用负荷缩放热设计功耗，延迟用roofline），WattGPU在未见模型-显卡组合的服务器场景下误差降低约4倍，在完全未见显卡时降低约2倍。WattGPU的数据与代码已公开于https://github.com/maufadel/wattgpu。
原文摘要 · Abstract (English)
Large Language Model (LLM) inference workloads are a rapidly growing contributor to data center energy consumption. Optimizing these deployments requires matching specific LLMs to the most efficient GPUs, but operators currently lack the tools to do so without exhaustively profiling each combination. While some predictive models exist, they still require profiling data and struggle to generalize to hardware unseen during training. To address this, we introduce \textit{WattGPU}, featuring two predictive models for mean GPU power draw and Inter-Token Latency (ITL). Our approach leverages only publicly available LLM metadata and GPU specifications, eliminating the need for hardware access or profiling while enabling generalization to unseen NVIDIA server-grade GPUs and LLMs. We evaluate our models using rigorous leave-one-GPU-out and leave-one-LLM-out cross-validation on a dataset of 42 open-source LLMs (0.1B--27B parameters) and 8 GPUs under both offline and server scenarios. The mean power draw model achieves a median absolute percentage error of $\leq3.4\%$ for offline and $\leq13.5\%$ for server scenarios on unseen GPUs, while the latency model achieves $\leq8.5\%$ in server mode, both maintaining strong GPU ranking correlations for server scenarios (Kendall $τ\geq0.76$). Compared to standard physically grounded baselines -- Load-Scaled Thermal Design Power (TDP) for power draw and roofline for latency -- our models reduce median absolute percentage error by approximately 4$\times$ on unseen LLM-GPU combinations for server scenarios or approximately 2$\times$ for completely unseen GPUs. WattGPU's data and code are publicly available at https://github.com/maufadel/wattgpu.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。