量化推理中,内存峰值可精准预测,且简单模型已足够。
Anatomy of a Quantized Agent: VRAM Stability and Forecasting in Code-Synthesis Agentic Workloads
- 用解析模型分解显存为权重、缓存和激活三部分,结合实测常数预测峰值显存。
- 在四个大模型上,解析模型误差仅2.2%-4.4%,优于复杂学习模型。
- 显存波动极小,适合代码生成的智能体无需复杂内存预测系统。
针对大模型推理中的峰值显存消耗,本文通过解析模型将其分解为权重存储、KV缓存和激活三项,参数依赖于步数、工具调用和上下文扩展。我们在一个受限实验中评估了该分解:基于LangGraph的CUDA内核合成智能体(AgentK)、4-bit量化系列(Q4 K M)、单张NVIDIA H100 GPU,以及四种LLM骨干网络,共1920条轨迹。聚焦峰值内存预测,发现两个核心结果:第一,当提供两个实测常数(加载权重显存和固定激活开销)时,解析模型在三个骨干网络上表现优于或匹配最佳学习基线(测试MAPE 2.2-4.4% vs. 3.4-6.5%,p=0.76);Phi-4-mini因显存方差极小(CV 0.3%),动态建模反而不如简单回归。第二,编译成功率严格依模型容量分界(从Phi-4-mini的5.7%到Qwen2.5-Coder-14B的62.0%),表明代码生成仍受模型内在能力限制而非显存。此外,所有骨干网络的峰值显存方差极低(CV 0.3%-9.4%),提示基于提示特征的学习回归相比常数均值基线无显著提升。因此,在高度量化、权重主导的场景下,部署复杂预测模型缺乏依据。论文发布完整数据集与匿名框架以支持复现。
原文摘要 · Abstract (English)
Analytical models of peak VRAM consumption for LLM inference decompose memory into weight-storage, KV-cache, and activation terms parameterized by step count, tool invocations, and context expansion. We evaluate this decomposition empirically within a strictly scoped measurement study: a LangGraph-based CUDA-kernel-synthesis agent (AgentK), a 4-bit quantization family (Q4 K M), a single NVIDIA H100 GPU, and four LLM backbones across 1,920 trajectories. Focusing on peak-memory forecasting behavior, we report two primary observations. First, closed-form analytical models achieve competitive accuracy when provided with two empirical constants: loaded-weight VRAM and a fixed activation-memory overhead. Supplied with live GPU readings and ground-truth trajectory parameters, the closed-form model matches or outperforms the best learned baseline on three of the four backbones (test MAPE 2.2-4.4% vs. 3.4-6.5%, p = 0.76). The exception is the smallest backbone (Phi-4-mini), where minimal VRAM variance (CV 0.3%) causes dynamic modeling to underperform simple regression. Second, compile success strictly bifurcates by backbone capacity (from 5.7% for Phi-4-mini to 62.0% for Qwen2.5-Coder-14B), demonstrating that functional code synthesis remains constrained by intrinsic LLM capabilities rather than available memory. Furthermore, because overall peak-memory variance is remarkably low across all backbones (CV 0.3-9.4%), learned prompt-feature regression offers statistically insignificant improvements over a constant-mean baseline. Consequently, we find no justification for deploying complex predictive VRAM models in highly quantized, weight-dominated regimes. We release the evaluated corpus and anonymized framework to support replication.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。