根据显存带宽动态切换批处理策略，提升大模型推理效率。
Threshold-Based Exclusive Batching for LLM Inference
- 基于显存带宽和负载比例，自动判断何时用独立批处理或混合批处理。
- 在带宽受限的显卡上，独立批处理最高提升41.9%吞吐量。
- 无需人工调参，可实时应对流量变化，适合生产环境部署。
混合批处理（MB）通过交错预填充和解码阶段，已成为大语言模型推理的标准调度策略，能有效提升计算与内存利用率。然而，通过受控实验发现，预填充与解码之间的干扰导致MB的每步边际成本高于纯解码。在高带宽的H200（4.8 TB/s）上，这一现象仅在解码令牌占比超过80%时出现；而在带宽受限的RTX PRO 6000（1.792 TB/s）上，该阈值骤降至20%。因此，最优选择取决于GPU内存带宽、模型规模及工作负载构成。本文推导出EB-MB性能交叉的闭式条件，并给出渐近最优的阶段切换阈值与内存安全的批大小设定。优化后的独立批处理（EB）在带宽受限设备上最高实现41.9%的吞吐量提升，而高带宽硬件上大模型仍更适合MB。提出的混合调度器EB+在线应用此条件，动态切换EB与MB，无需人工干预。在非平稳流量场景下（分布或并发性变化），EB+在所有设置中均达到最高或接近最高吞吐量，相比MB最高提升36.4%。
原文摘要 · Abstract (English)
Mixed batching (MB)--interleaving prefill and decode in a single batch--has become the standard scheduling strategy for large language model (LLM) inference due to its efficiency in maximizing compute and memory utilization. However, through controlled experiments, we find that prefill-decode interference inflates MB's per-step marginal cost above that of pure decode. On the high-bandwidth H200 (4.8 TB/s), this occurs only when decode tokens exceed 80% of the batch; however, on the bandwidth-constrained RTX PRO 6000 (1.792 TB/s), this threshold plummets to just 20%. Consequently, the optimal choice between MB and exclusive batching (EB) fundamentally depends on GPU memory bandwidth, model size, and workload composition. We derive a closed-form condition for this EB-MB performance crossover, along with asymptotically optimal phase-switching thresholds and memory-safe batch sizing for EB. Optimized EB achieves up to 41.9% higher throughput on bandwidth-constrained GPUs, while MB retains its advantage on high-bandwidth hardware with larger models. Our hybrid scheduler EB+ applies this condition online to dynamically switch between EB and MB without manual intervention. Under non-stationary traffic with distribution or concurrency shifts, EB+ attains the highest or near-highest throughput in every setting, outperforming MB by up to 36.4%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。