实时追踪多租户大模型推理延迟，精准分配每请求资源消耗
LLMVisor: A Real-Time Latency Attribution Model for Multi-Tenant LLM Serving
- 基于算力与内存流量的分段线性模型，分解批量请求延迟
- 预填充和解码阶段误差降低2.5倍以上，99%分位性能提升4.4倍
- 微秒级运行效率，适合嵌入调度循环，支持细粒度资源共享
随着大模型推理转向多租户GPU集群，共批处理虽提升吞吐量，但掩盖了各租户的使用情况并限制控制能力。实现推理引擎的细粒度共享需一个实时、精确且轻量的请求级延迟归因机制。本文提出LLMVisor，一种基于屋顶线模型的延迟归因方法，通过与浮点运算量（FLOPs）和内存读写流量成比例的特征，捕捉计算密集和内存受限阶段，以简洁的分段线性形式建模。该方法将批次延迟分解为可加的单请求份额，并在微秒级内完成计算。我们在A100/H100 GPU上对Llama 3.1-8B及Qwen 2.5-14B/32B在不同张量并行度和负载组合下进行了评估。相比基于标记数的基线，LLMVisor在预填充阶段的p90和p99相对误差分别减少2.5倍和3.3倍，在解码阶段分别减少3.5倍和4.4倍，即使面对批处理变化和序列差异仍保持高精度。
原文摘要 · Abstract (English)
As LLM inference shifts to multi-tenant GPU clusters, co-batching improves throughput but obscures per-tenant usage and limits control. Enabling fractional sharing of the inference engine requires a real-time, per-request attribution primitive that is accurate and light enough to run inside the scheduling loop. We present LLMVisor, a roofline-guided latency attribution model that captures the memory-bound and compute-bound phases via a concise piecewise-linear form over features proportional to FLOPs and memory I/O traffic. LLMVisor decomposes batch latency into additive, per-request shares and runs efficiently at microsecond scale. We evaluate LLMVisor across Llama 3.1-8B and Qwen 2.5-14B/32B on A100/H100 GPUs under varying tensor parallelism and workload mixes. Compared to a token-count baseline, LLMVisor attains near-perfect R-squared and reduces relative error by up to 2.5x and 3.3x at p90 and p99, respectively, for prefill, and by up to 3.5x and 4.4x for decode, despite batching variability and sequence divergence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。