发现大模型服务中人类感知延迟要求导致吞吐量损失60%-93%。
Human-Less LLM Serving: Quantifying the Human Tax on Throughput
- 针对无感任务设计无人类延迟约束的最优服务模式
- 长上下文下吞吐量损失高达60%-93%
- 适合高并发、长文本生成等程序化任务场景
主流大模型服务系统均以满足TTFT和TPOT SLO为目标，这些指标反映的是人类用户感知的延迟，相关机制已成为标准基础设施。我们发现，长周期AI任务会以程序化方式在紧密循环中调用大模型，而无人类观察到TTFT或TPOT。于是提出问题：这些无需人类感知延迟的任务，为何仍需牺牲大量吞吐量以满足其SLO？我们对分块大小、SLO设置、上下文长度和并发级别进行了系统性测量。结果表明，人类延迟税随上下文长度显著上升，在64K token时达到60%-93%区间；将TTFT SLO收紧至生产标准，吞吐量损失远超无约束基线。该损耗在高并发下更明显，且在SGLang与Sarathi-Serve中表现相似。我们定义了无约束的人类缺失最优服务，并实现原型验证其在真实任务中的可行性。研究建议服务系统应支持按工作负载类型配置SLA，而非对所有流量统一施加人类延迟税。
原文摘要 · Abstract (English)
Every major LLM serving system is designed to meet TTFT and TPOT SLOs. These metrics capture latency as a human user perceives it, and the mechanisms built to satisfy them are now standard infrastructure. We observe that long-horizon AI tasks call LLMs programmatically in tight loops where no human observes TTFT or TPOT. We ask: how much throughput do serving systems sacrifice to meet TTFT and TPOT SLAs that these workloads never need? We conduct a systematic measurement study across chunk sizes, SLO settings, context lengths, and concurrency levels. We find that the human tax on throughput grows substantially with context length and lands in the 60-93% range. At 64K token contexts, tightening the TTFT SLO to production-typical settings costs a large fraction of throughput versus the human-less baseline. The human tax is larger at higher concurrency and is qualitatively similar across SGLang and Sarathi-Serve. We term the unconstrained optimum human-less serving and provide a prototype demonstrating that it is practical on real workloads. Our findings argue that serving systems should expose workload-class-aware SLA configurations rather than silently applying the human tax uniformly to all traffic.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。