通过规整KV缓存移动,让静态图推理更高效稳定。
KV-RM: Regularizing KV-Cache Movement for Static-Graph LLM Serving

- 将逻辑缓存与物理存储解耦,用块页管理器跟踪活跃状态
- 合并不连续缓存映射为大块传输,提升注意力核效率
- 在真实负载下降低延迟峰值,减少内存占用,适合生产部署
静态图LLM解码器具有可预测的启动、固定张量形状和低提交开销,但在线解码时面临高度不规则的KV缓存行为:请求长度各异、结束事件异步到达、逻辑历史随时间碎片化。动态运行时通过分页KV管理和步级调度恢复灵活性,而静态图执行器常过度预留内存并出现突发延迟尖峰。本文探讨是否可将此类波动吸收在固定解码接口之下。提出KV-RM,一种在静态图解码器下方规整KV缓存移动的运行时设计。该设计将逻辑KV历史与物理存储解耦,通过块页管理器追踪活跃状态,并以单一已提交描述符实现每步解码。合并式传输路径在固定形状注意力核消费前,将非连续的KV映射聚合成少量大传输组。可在同一接口下启用有界远历史摘要,但核心设计不依赖于此。在2个NVIDIA A100 GPU节点上,相比静态图基线,KV-RM提升了混合长度解码吞吐量与尾部延迟表现,降低了各类工作负载下的保留KV内存,并在生产级轨迹回放中消除了严重的突发延迟尖峰。结果表明,在静态图服务中,将缓存移动而非内核形状作为灵活性边界,是有效的。
原文摘要 · Abstract (English)
Static-graph LLM decoders provide predictable launches, fixed tensor shapes, and low submission overhead, but online decoding exposes highly irregular KV-cache behavior: request lengths differ, EOS events arrive asynchronously, and logical histories fragment over time. Dynamic runtimes recover flexibility through paged KV management and step-level scheduling, while static-graph executors often over-reserve memory and suffer burst-time latency outliers. This paper studies whether much of this variability can be absorbed below a fixed decode interface. We present KV-RM, a runtime design that regularizes KV-cache movement beneath a static-graph LLM decoder. KV-RM decouples logical KV histories from physical storage, tracks active KV state through a block pager, and materializes each decode step through a single committed descriptor. A merge-staged transport path coalesces non-contiguous KV mappings into a small number of large transfer groups before a fixed-shape attention kernel consumes them. Optional bounded far-history summaries can be enabled under the same interface, but the core design does not depend on them. On a 2-GPU NVIDIA A100 node, KV-RM improves mixed-length decoding throughput and tail latency relative to a static-graph baseline, reduces reserved KV memory across workload families, and removes severe burst-time latency spikes under production-trace replay. These results suggest that KV-cache movement, rather than kernel shape, can be an effective boundary for recovering runtime flexibility in static-graph LLM serving.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。