通过错峰调度提升大模型推理速度与吞吐，降低首令牌延迟30%-40%。
Staggered Batch Scheduling: Co-optimizing Time-to-First-Token and Throughput for High-Efficiency LLM Inference
- 错峰批量调度：主动缓冲请求，避免内部排队和并行空档。
- 首令牌延迟降低30%-40%，吞吐量提升15%-20%。
- 适合高并发大模型服务场景，尤其适用于分布式推理架构。
大型语言模型（LLM）服务正向复杂的分布式架构演进，特别是分离参数与数据的DP+EP大规模部署模式，带来了独特的调度挑战。与传统部署中可将实例视为黑盒不同，DP+EP架构存在较高的内部同步开销。我们发现，立即调度请求会导致严重的内部排队和并行空档，从而恶化首令牌延迟（TTFT）。为此，我们提出错峰批量调度（SBS），通过主动缓冲请求形成最优执行批次，实现时间解耦，消除内部排队空档，同时不牺牲吞吐。进一步地，利用缓冲带来的调度窗口，我们引入负载感知全局分配策略，在预填充和解码阶段均衡分配计算负载至各DP单元。在生产级H800集群上部署Deepseek-V3，相较最先进的即时调度基线，系统将TTFT降低30%-40%，吞吐提升15%-20%。
原文摘要 · Abstract (English)
The evolution of Large Language Model (LLM) serving towards complex, distributed architectures--specifically the P/D-separated, large-scale DP+EP paradigm--introduces distinct scheduling challenges. Unlike traditional deployments where schedulers can treat instances as black boxes, DP+EP architectures exhibit high internal synchronization costs. We identify that immediate request dispatching in such systems leads to severe in-engine queuing and parallelization bubbles, degrading Time-to-First-Token (TTFT). To address this, we propose Staggered Batch Scheduling (SBS), a mechanism that deliberately buffers requests to form optimal execution batches. This temporal decoupling eliminates internal queuing bubbles without compromising throughput. Furthermore, leveraging the scheduling window created by buffering, we introduce a Load-Aware Global Allocation strategy that balances computational load across DP units for both Prefill and Decode phases. Deployed on a production H800 cluster serving Deepseek-V3, our system reduces TTFT by 30%-40% and improves throughput by 15%-20% compared to state-of-the-art immediate scheduling baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。