发现大批次推理中内存瓶颈才是主因，而非计算瓶颈。
Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inference
- 通过GPU级分析揭示大批次推理受内存带宽限制
- 提出配置优化器减少显存占用，提升资源利用率
- 适合关注小模型部署与多任务并行的工程师
大规模语言模型虽广泛应用，但其自回归生成特性导致推理时资源利用效率低。尽管批量处理可提升吞吐量，但超过一定规模后性能增长停滞，尤其在小模型上明显，现有研究通常归因于进入计算瓶颈阶段。本文通过深入的GPU层级分析发现，大批次推理仍处于内存瓶颈状态，主要受限于DRAM带宽饱和，而大部分GPU计算能力未被充分利用。为此，我们提出批处理配置顾问（BCA），通过优化内存分配降低显存需求，对吞吐量影响极小。释放出的显存与闲置计算能力可用于并发负载，例如通过模型复制提升服务吞吐量与GPU利用率。研究结果挑战了传统认知，为小模型推理的资源优化提供了新思路。代码已开源：https://github.com/FerranAgulloLopez/vLLMBatchingMemoryGap。
原文摘要 · Abstract (English)
Large language models have been widely adopted across different tasks, but their auto-regressive generation nature often leads to inefficient resource utilization during inference. While batching is commonly used to increase throughput, performance gains plateau beyond a certain batch size, especially with smaller models, a phenomenon that existing literature typically explains as a shift to the compute-bound regime. In this paper, through an in-depth GPU-level analysis, we reveal that large-batch inference remains memory-bound, with most GPU compute capabilities underutilized due to DRAM bandwidth saturation as the primary bottleneck. To address this, we propose a Batching Configuration Advisor (BCA) that optimizes memory allocation, reducing GPU memory requirements with minimal impact on throughput. The freed memory and underutilized GPU compute capabilities can then be leveraged by concurrent workloads. Specifically, we use model replication to improve serving throughput and GPU utilization. Our findings challenge conventional assumptions about LLM inference, offering new insights and practical strategies for improving resource utilization, particularly for smaller language models. The code is publicly available at https://github.com/FerranAgulloLopez/vLLMBatchingMemoryGap.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。