用GPU原生视频编码实现高效远程键值缓存复用，加速大模型推理。
Efficient Remote KV Cache Reuse with GPU-native Video Codec
- 利用GPU视频编码技术压缩键值缓存，提升传输效率。
- 实测将首次输出时间缩短至原来的28.5%，且无精度损失。
- 适合高延迟低带宽场景下部署的大模型服务系统。
远程键值（KV）缓存复用从远程存储中获取相同上下文的缓存数据，避免重复计算，从而加速大语言模型推理。尽管在高速网络中表现优异，但在带宽受限场景下性能显著下降。现有方法通过压缩传输KV缓存来缓解该问题，但解压开销抵消了复用优势。本文提出一种高效且可广泛部署的远程KV缓存复用方案——KVCodec，其核心在于利用GPU原生视频编码器。该系统通过两种技术实现：一是编码友好的张量布局，将KV缓存转换为紧凑的视频格式，支持快速传输；二是高效的缓存获取器，以流水线方式协同传输、解码与恢复过程，消除资源竞争，隐藏网络波动影响，实现最小化首次输出时间（TTFT）。我们在多款从高端到低端的GPU上实现了原型系统。实验表明，相比当前最优方法，其可将首次输出时间缩短最多达3.51倍，且保持无损精度。
原文摘要 · Abstract (English)
Remote KV cache reuse fetches KV cache for identical contexts from remote storage, avoiding recomputation, accelerating LLM inference. While it excels in high-speed networks, its performance degrades significantly in bandwidth-limited scenarios. Recent studies address this by transmitting KV caches in compressed form, but the associated heavyweight decompression counteracts the KV reuse benefits. In this paper, we propose an efficient and widely deployable remote KV cache reuse solution that leverages GPU-native video codecs. Our system, KVCodec, enables effective KV cache coding with two techniques. The codec-friendly tensor layout compresses the KV cache in a highly compact video format, enabling fast transmission. The efficient KV fetcher orchestrates the transmission, decoding, and restoration of compressed KV caches in an efficient pipelined manner, eliminating resource contention, masking network fluctuations, and achieving minimum time-to-first-token (TTFT). We prototype KVCodec on diverse GPUs from high- to low-end. Experiments reveal that it reduces TTFT by up to 3.51 times while maintaining lossless accuracy, compared to SOTA methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。