根据请求长度智能分配模型实例，显著降低大语言模型推理成本。
Token-Budget-Aware Pool Routing for Cost-Efficient LLM Inference
- 按请求的总词元预算动态分配至短文本或长文本专用池
- 在真实流量下节省17%-39%的GPU实例，年省百万美元
- 无需分词器、零额外开销，适合生产环境部署
生产级vLLM集群通常按最坏情况配置上下文长度，导致80%-95%的短请求浪费4-8倍并发能力，同时引发KV缓存失败——内存溢出、抢占风暴和请求拒绝。两者根源均为配置与流量不匹配。本文提出基于词元预算的池路由：通过自校准的类别级每字节词元比估算每个请求的总词元预算，将其分配至高吞吐短池或高容量长池，两类池均按工作负载特性精准配置。该比率通过提示词中的词元数反馈，采用指数移动平均在线学习，无需分词器。基于两个可观测量（短请求占比alpha与吞吐增益比rho）的闭式成本模型预测整体节省：savings = alpha * (1 - 1/rho)。在Azure LLM推理数据集和LMSYS-Chat-1M上对Llama-3-70B部署于A100 GPU的实测显示，该方法可减少17%-39%的GPU实例（1,000请求/秒下年省120万-200万美元），并通过自包含离散事件模拟验证。案例研究显示，若将Qwen3-235B-A22B部署于AMD MI300X，10,000请求/秒下年节省达1540万美元。该算法仅增加O(1)调度开销，跨内容类型自校准，且兼容PagedAttention、连续批处理与预填充-解码解耦。
原文摘要 · Abstract (English)
Production vLLM fleets provision every instance for worst-case context length, wasting 4-8x concurrency on the 80-95% of requests that are short and simultaneously triggering KV-cache failures -- OOM crashes, preemption storms, and request rejections. Both problems share a single root cause: configuration-traffic mismatch. We propose token-budget-aware pool routing: estimate each request's total token budget using a self-calibrating per-category bytes-per-token ratio, then dispatch it to one of two vLLM pools -- a high-throughput short pool or a high-capacity long pool -- each right-sized for its workload class. The ratio is learned online via exponential moving average from usage.prompt_tokens feedback, requiring no tokenizer. A closed-form cost model, savings = alpha * (1 - 1/rho), predicts fleet-level GPU savings from two observable quantities: the short-traffic fraction alpha and the throughput gain ratio rho. On traces from the Azure LLM Inference Dataset and LMSYS-Chat-1M serving Llama-3-70B on A100 GPUs, token-budget routing reduces GPU instances by 17-39% (\$1.2-2.0M/yr at 1,000 req/s), with savings verified by a self-contained discrete-event simulator. A case study projecting Qwen3-235B-A22B on AMD MI300X at 10,000 req/s shows \$15.4M/yr in savings. The algorithm adds O(1) dispatch overhead, self-calibrates across content types without a tokenizer, and composes with PagedAttention, continuous batching, and prefill-decode disaggregation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。