通过资源感知调度，让大模型离线推理吞吐提升44%。
BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batching
- 用资源感知前缀树协调请求重排与重叠，兼顾计算共享和内存复用。
- 在多模态负载下，相比vLLM和SGLang，吞吐最高提升1.44倍。
- 适合对延迟不敏感的大规模离线推理任务，尤其多模态场景。
离线批处理推理利用请求批量化提升吞吐并降低成本，日益成为对延迟不敏感应用的主流方案。然而，随着模型能力与模态多样化，请求的计算与内存需求差异增大，为资源重叠带来新机会。但最大化资源重叠的调度策略可能与广泛使用的前缀共享策略冲突，导致推理吞吐次优。本文提出BlendServe系统，通过资源感知前缀树，同时实现资源重叠与前缀共享的优势。该系统利用离线推理中宽松的延迟要求，重新排序并重叠不同资源需求的请求，同时保持高前缀共享率。我们在多种合成多模态工作负载上评估BlendServe，结果表明其相比业界标准vLLM和SGLang，吞吐最高提升1.44倍。
原文摘要 · Abstract (English)
Offline batch inference, which leverages the flexibility of request batching to achieve higher throughput and lower costs, is becoming more popular for latency-insensitive applications. Meanwhile, recent progress in model capability and modality makes requests more diverse in compute and memory demands, creating unique opportunities for throughput improvement by resource overlapping. However, a request schedule that maximizes resource overlapping can conflict with the schedule that maximizes prefix sharing, a widely-used performance optimization, causing sub-optimal inference throughput. We present BlendServe, a system that maximizes resource utilization of offline batch inference by combining the benefits of resource overlapping and prefix sharing using a resource-aware prefix tree. BlendServe exploits the relaxed latency requirements in offline batch inference to reorder and overlap requests with varied resource demands while ensuring high prefix sharing. We evaluate BlendServe on a variety of synthetic multi-modal workloads and show that it provides up to $1.44\times$ throughput boost compared to widely-used industry standards, vLLM and SGLang.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。