通过周期性路由提升大模型服务缓存命中率，兼顾负载均衡与性能。
CacheRoute: Planned Prefix-Affinity Routing for Large-Scale LLM Serving
- 设计周期性路由计划，动态分配请求以保持前缀缓存亲和性。
- 在60张H100上实现176 QPS、p99延迟3.5秒，缓存命中率从64%升至93%。
- 推荐部署前用影子回放验证，避免亲和性带来的负载不均问题。
前缀缓存仅在重复请求返回仍持有前缀键值对的服务器时才有效。无缓存感知的负载均衡会分散复用机会；固定亲和性虽能保留缓存但可能导致服务器过载。CacheRoute通过周期性路由规划解决此权衡：将高频率访问键引入稳定热集，并根据预期负载分配其目标。热键可映射至多个目的地，但在主半合成数据集中每个键仅对应一个。在60块H100 GPU上运行Llama-3.3-70B（fp8精度）时，CacheRoute维持176±11 QPS，满足3.5秒的p99 SLO，是五种基线中最强者2.3倍；服务端键值缓存命中率从64.1±1.3%提升至93.2±0.5%。第二个半合成数据集及受控的8B和突发实验分离了亲和性与分配策略的影响。两个32B工作负载显示：当亲和性恢复的缓存工作量过少时，残余负载偏差会削弱甚至抵消性能增益。因此建议部署前通过影子回放验证，而非仅依赖工作负载统计启用亲和性。
原文摘要 · Abstract (English)
Prefix caching avoids prefill only when a repeated request returns to a server that still holds the prefix KV. Cache-blind balancing disperses that reuse; fixed affinity preserves it but can overload a server. CacheRoute resolves this tradeoff with a periodic routing plan. It admits high-rate keys to a stable warm set and places their assignments by expected load. Hot keys may use more than one destination, although every key in our primary semi-synthetic aggregate uses exactly one. On Llama-3.3-70B in fp8 across 60 H100 GPUs, CacheRoute sustains 176+/-11 QPS at a 3.5-s p99 SLO, 2.3x the strongest of five baselines. Served KV-cache hit rate rises from 64.1+/-1.3% under cache-blind balancing to 93.2+/-0.5%. A second semi-synthetic aggregate and controlled 8B and burst experiments separate the effects of affinity and placement. Two 32B workloads provide the counterexamples: when affinity recovers too little KV work, its residual load skew reduces or erases the improvement. We therefore recommend gating any deployment with a shadow replay rather than enabling affinity from workload statistics alone.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。