对比vLLM与HuggingFace TGI，发现前者吞吐高24倍，后者延迟更低。
Comparative Analysis of Large Language Model Inference Serving Systems: A Performance Study of vLLM and HuggingFace TGI
- vLLM用分页注意力机制提升吞吐，TGI优化单用户交互延迟。
- 70B模型下vLLM吞吐达TGI的24倍，但TGI尾延迟更低。
- 高并发选vLLM，低延迟互动场景选TGI。
大型语言模型（LLMs）在生产环境中的部署需要高效推理服务系统以平衡吞吐量、延迟和资源利用率。本文对两个主流开源LLM服务框架vLLM和HuggingFace Text Generation Inference（TGI）进行了全面实证评估。我们使用7B至70B参数的LLaMA-2模型，在吞吐性能、端到端延迟、GPU内存利用率和可扩展性等多个维度进行基准测试。实验表明，vLLM通过其创新的PagedAttention机制，在高并发工作负载下实现比TGI高出最多24倍的吞吐量；而TGI在交互式单用户场景中表现出更低的尾延迟。研究提供了不同部署场景下的详细性能分析，并根据工作负载特性提出实用选型建议：vLLM适用于高吞吐批量处理，TGI更适合中等并发下的低延迟交互应用。
原文摘要 · Abstract (English)
The deployment of Large Language Models (LLMs) in production environments requires efficient inference serving systems that balance throughput, latency, and resource utilization. This paper presents a comprehensive empirical evaluation of two prominent open-source LLM serving frameworks: vLLM and HuggingFace Text Generation Inference (TGI). We benchmark these systems across multiple dimensions including throughput performance, end-to-end latency, GPU memory utilization, and scalability characteristics using LLaMA-2 models ranging from 7B to 70B parameters. Our experiments reveal that vLLM achieves up to 24x higher throughput than TGI under high-concurrency workloads through its novel PagedAttention mechanism, while TGI demonstrates lower tail latencies for interactive single-user scenarios. We provide detailed performance profiles for different deployment scenarios and offer practical recommendations for system selection based on workload characteristics. Our findings indicate that the choice between these frameworks should be guided by specific use-case requirements: vLLM excels in high-throughput batch processing scenarios, while TGI is better suited for latency-sensitive interactive applications with moderate concurrency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。