根据请求特点动态选最优推理模式,显著降低单卡大模型延迟和能耗。
RequestRouter: Request-Boundary Routing for Efficient Single-GPU LLM Inference
- 按请求特征选择固定推理模式,如量化、预缓存、推测解码等
- 平均延迟提速2.1倍,能耗降至48%,精度损失仅0.4%
- 轻量路由开销极小,适合实际服务部署
RequestRouter 是一种轻量级请求边界控制器,用于降低单卡大语言模型推理的延迟和能耗。不同于使用单一静态配置处理所有请求,该系统利用低成本的请求级特征,为每个请求选择一种固定的推理模式,包括FP16、量化推理、推测解码、前缀缓存、连续批处理,以及混合模式(如GPTQ+前缀缓存、INT8+连续批处理)。我们在NVIDIA A100 GPU上通过vLLM部署一个80亿参数的指令微调模型进行评估。全规模A100测试包含26,500次固定模式评估和3,500次在线控制器评估,共30,000次推理执行:相比FP16,系统实现2.10倍平均延迟加速,能效比达0.48。小规模重复测量验证结果稳定:延迟提速1.93倍(95%置信区间:1.88–1.98倍),能效比0.523(95%置信区间:0.506–0.540)。在扩展自动基准测试中,路由策略保持99.6%的FP16宏准确率。10万次CPU微基准测试显示,平均路由开销仅0.00475毫秒(p99为0.00532毫秒)。因此,简单的请求感知路由可在不重训练或修改底层LLM的前提下,大幅提升推理效率。
原文摘要 · Abstract (English)
RequestRouter is a lightweight request-boundary controller for reducing the latency and energy cost of single-GPU large language model inference. Rather than serving all requests with one static configuration, the system uses cheap request-level features to select one fixed inference mode per request, including FP16, quantized inference, speculative decoding, prefix caching, continuous batching, and hybrid modes such as GPTQ plus prefix caching and INT8 plus continuous batching. We evaluate RequestRouter using an 8B instruction-tuned language model served through vLLM on NVIDIA A100 GPUs. Across the full-scale A100 evaluation--26,500 fixed-mode evaluations followed by 3,500 online-controller evaluations, for 30,000 measured inference executions in total--the controller achieves a 2.10x mean latency speedup over FP16 and a 0.48x energy ratio on deployment-style workloads. A smaller matched evaluation with repeated measurements provides a controlled statistical check of this result: RequestRouter retains a 1.93x latency speedup (95% CI: 1.88--1.98x) and a 0.523 energy ratio (95% CI: 0.506--0.540), showing that the gains persist under a more tightly controlled protocol. On a separate expanded automatic benchmark evaluation, the routed policy retains 99.6% of FP16 macro accuracy. A 100,000-call CPU microbenchmark measures only 0.00475 ms mean routing overhead (0.00532 ms p99). Thus, simple request-aware routing can recover substantial serving efficiency without retraining or modifying the underlying LLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。