按预测生成时长分组请求，提升大模型推理吞吐量
Multi-Bin Batching for Increasing LLM Inference Throughput
- 将相似生成时长的请求分入预设桶中批量处理
- 实测吞吐量显著高于静态批处理，接近连续批处理水平
- 适合追求高吞吐的生产环境部署
随着大语言模型应用日益广泛，提升其推理系统的效率变得愈发关键。批处理是服务器（如GPU）调度推理任务的核心环节，通过并行处理多个请求以最大化吞吐量。然而，请求生成长度差异大，导致硬件需等待最长请求完成才能进入下一阶段，造成资源浪费。本文从排队论角度建模该问题，提出多桶批处理（Multi-Bin Batching）方法：将具有相似（预测）执行时间的请求分入预设的若干桶中。结合理论分析与实验，包括真实场景下的静态和连续批处理基线，验证了该方法在静态批处理框架下可显著提升吞吐量，并量化了其与原生连续批处理之间的差距，无论使用理想还是估计的生成长度信息。
原文摘要 · Abstract (English)
As large language models (LLMs) grow in popularity for their diverse capabilities, improving the efficiency of their inference systems has become increasingly critical. Batching LLM requests is a critical step in scheduling the inference jobs on servers (e.g. GPUs), enabling the system to maximize throughput by allowing multiple requests to be processed in parallel. However, requests often have varying generation lengths, causing resource underutilization, as hardware must wait for the longest-running request in the batch to complete before moving to the next batch. We formalize this problem from a queueing-theoretic perspective, and aim to design a control policy which is throughput-optimal under a static-batching framework. We propose Multi-Bin Batching, a simple yet effective method that can provably improve LLM inference throughput under this framework by grouping requests with similar (predicted) execution times into predetermined bins. Through a combination of theoretical analysis and experiments, including real-world LLM inference scenarios with static and continuous-batching baselines, we demonstrate that multi-bin batching substantially improves throughput over static batching and quantify the remaining gap to native continuous batching under both oracle and estimated length information.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。