AccelGen提升大模型推理吞吐，支持长短提示与不同延迟要求。
AccelGen: Heterogeneous SLO-Guaranteed High-Throughput LLM Inference Serving for Diverse Applications
- 动态分块调节大小，兼顾计算效率与响应延迟
- 按请求延迟需求优先级批处理，提升资源利用率
- 同时优化显卡算力与缓存使用，逼近最优性能
本文针对支持多种应用的大语言模型（LLM）推理服务系统中混合短/长提示、异构迭代延迟目标（SLO）的场景，提出AccelGen高吞吐推理系统。该系统引入四个核心组件：（1）基于SLO的动态分块，动态调整分块大小以最大化GPU算力利用率并满足迭代级SLO；（2）基于迭代级SLO的任务优先级调度，优先处理紧SLO请求，并对相似SLO请求进行批处理；（3）多资源感知批处理，选择队列中的请求以最大化GPU算力和键值缓存（KVC）利用率。基于真实流量轨迹的实验表明，AccelGen相比当前最先进方法实现1.42–11.21倍更高吞吐、1.43–13.71倍更高有效吞吐、37%–90%更高的SLO达标率，以及1.61–12.22倍更低响应延迟，性能接近理想优化器（Oracle）水平。
原文摘要 · Abstract (English)
In this paper, we consider a mixed-prompt scenario for a large language model (LLM) inference serving system that supports diverse applications with both short prompts and long prompts and heterogeneous SLOs for iteration time. To improve throughput when handling long prompts, previous research introduces a chunking method, but has not addressed heterogeneous SLOs. To address the limitation, we propose AccelGen, a high-throughput LLM inference serving system with heterogeneous SLO guarantees for diverse applications. AccelGen introduces four core components: (1) SLO-guaranteed dynamic chunking, which dynamically adjusts chunk sizes to maximize GPU compute utilization while meeting iteration-level SLOs; (2) Iteration-level SLO-based task prioritization, which prioritizes tight-SLO requests and batches requests with similar SLOs; (3) Multi-resource-aware batching, which selects queued requests to maximize the utilizations of both GPU compute resource and key-value cache (KVC). Trace-driven real experiments demonstrate that AccelGen achieves 1.42-11.21X higher throughput, 1.43-13.71X higher goodput, 37-90% higher SLO attainment, and 1.61-12.22X lower response latency compared to the state-of-the-art approaches. It achieves performance near the Oracle, which optimally maximizes goodput.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。