通过预占调度提升大模型流式生成响应速度
TokenFlow: Responsive LLM Text Streaming Serving under Request Burst via Preemptive Scheduling
- 动态优先级调度+预判缓存迁移，应对突发请求
- 实测用户感知吞吐提升82.5%，首字延迟降低80.2%
- 适合高并发流式文本生成场景，如聊天机器人
实时大模型交互需要流式输出，即逐步生成并传输文本令牌，同时兼顾响应速度（首令牌延迟低）和生成稳定性（令牌间隔稳定）。传统服务系统因非抢占式调度和被动内存管理，导致资源利用率低、突发请求下并行处理能力差。为此，我们提出TokenFlow，一种基于抢占式调度与主动键值缓存管理的新型大模型服务系统。该系统根据实时令牌缓冲区占用率和消费速率动态调整请求优先级，后台主动在GPU与CPU间迁移KV缓存，并重叠I/O与计算以减少抢占开销。在多个GPU（RTX 4090、A6000、H200）上对Llama3-8B和Qwen2.5-32B的实验表明，TokenFlow在不降低整体令牌吞吐的前提下，有效吞吐量（按实际用户消费计算）最高提升82.5%，P99首令牌延迟最多降低80.2%。
原文摘要 · Abstract (English)
Real-time LLM interactions demand streamed token generations, where text tokens are progressively generated and delivered to users while balancing two objectives: responsiveness (i.e., low time-to-first-token) and steady generation (i.e.,required time-between-tokens). Standard LLM serving systems suffer from the inflexibility caused by non-preemptive request scheduling and reactive memory management, leading to poor resource utilization and low request processing parallelism under request bursts. Therefore, we present TokenFlow, a novel LLM serving system with enhanced text streaming performance via preemptive request scheduling and proactive key-value (KV) cache management. TokenFlow dynamically prioritizes requests based on real-time token buffer occupancy and token consumption rate, while actively transferring KV cache between GPU and CPU memory in the background and overlapping I/O with computation to minimize request preemption overhead. Extensive experiments on Llama3-8B and Qwen2.5-32B across multiple GPUs (RTX 4090, A6000, H200) demonstrate that TokenFlow achieves up to 82.5% higher effective throughput (accounting for actual user consumption) while reducing P99 TTFT by up to 80.2%, without degrading overall token throughput.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。