通过调度与内存协同设计，显著降低大模型在线服务的响应延迟。
PRISM: Fast Online LLM Serving via Scheduling-Memory Co-design
- 联合优化调度与缓存，根据请求特征动态保留高频前缀
- 在4B和13B模型上分别将P99延迟降低23.3%和37.1%
- 适合高并发、存在热点内容的实时大模型服务场景
现代在线大语言模型服务（如RAG和智能体系统）普遍存在提示分段和热点偏斜现象：少量提示片段在用户请求中频繁重复。若不联合利用这些模式，会导致热点片段反复预填充，延长首字响应时间（TTFT），影响吞吐和用户体验。现有工作独立处理缓存与调度，未能对齐请求接入与缓存保留策略。本文分析调度与缓存协同对TTFT的影响，提出PRISM（Prefix Reuse Optimization Integrated Scheduling and Memory），通过查询感知调度器（QAS）与需求感知基数树（DART）实现请求接入与精确前缀缓存保留的一致性。评估显示，相较于最强基线，PRISM在4B和13B模型上分别降低平均每QPS的P99 TTFT 23.3%和37.1%，并提升精确前缀缓存命中率5.9和12.2个百分点。
原文摘要 · Abstract (English)
Modern online large language model (LLM) services, such as Retrieval-Augmented Generation (RAG) and agent systems, increasingly expose two prominent characteristics: prompt segmentation (e.g., system instructions, retrieved passages, tool outputs) and hotspot skew, where a small set of these segments recurs frequently across user requests. Failing to jointly exploit these patterns could lead to repeated prefill of hot segments and prolonged TTFT, undermining both throughput and user-perceived responsiveness. However, existing work tackles these patterns independently: KV-cache management mainly exploits segment reuse while scheduling reorders requests to improve cache locality, yet neither aligns request admission with KV-cache retention. To address this gap, we first analyze how scheduling and KV-cache management jointly affect TTFT. Guided by this, we present PRISM (Prefix Reuse Optimization Integrated Scheduling and Memory), which co-designs a query-aware scheduler (QAS) with a demand-aware radix tree (DART) to align request admission with exact-prefix KV retention. Our evaluation results show that, versus the strongest baseline, PRISM reduces average per-QPS P99 TTFT by 23.3\% and 37.1\% while increasing exact-prefix KV-cache hit rate by 5.9 and 12.2 percentage points on 4B and 13B models, respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。