ParisKV提升长文本大模型推理速度与稳定性，解决缓存检索漂移和延迟问题。
ParisKV: Fast and Drift-Robust KV-Cache Retrieval for Long-Context LLMs
- 基于哈希碰撞筛选候选，结合量化内积重排，实现高效缓存检索
- 百万级上下文下推理速度超全注意力2.8倍，延迟降低17至44倍
- 支持CPU离线缓存，适合大规模长文本生成场景
KV缓存检索对长上下文大模型推理至关重要，但现有方法在规模下易受分布漂移影响且延迟高。我们提出ParisKV，一种基于哈希碰撞候选选择、后接量化内积重排估计的抗漂移GPU原生检索框架。针对百万级令牌上下文，ParisKV通过统一虚拟地址（UVA）支持CPU离线缓存，实现按需获取top-$k$缓存，开销极低。在长输入与长生成基准测试中，ParisKV性能达或超过全注意力水平；其解码效率为当前最优：即使在批量大小为1时也匹配或超越全注意力速度，在全注意力可运行范围内吞吐量最高提升2.8倍，并可扩展至全注意力内存溢出的百万级上下文。在百万级规模下，相比MagicPIG和PQCache两大先进基线，解码延迟分别降低17倍和44倍。代码已开源：https://github.com/amy-77/ParisKV/tree/main。
原文摘要 · Abstract (English)
KV-cache retrieval is essential for long-context LLM inference, yet existing methods struggle with distribution drift and high latency at scale. We introduce ParisKV, a drift-robust, GPU-native KV-cache retrieval framework based on collision-based candidate selection, followed by a quantized inner-product reranking estimator. For million-token contexts, ParisKV supports CPU-offloaded KV caches via Unified Virtual Addressing (UVA), enabling on-demand top-$k$ fetching with minimal overhead. ParisKV matches or outperforms full attention quality on long-input and long-generation benchmarks. It achieves state-of-the-art long-context decoding efficiency: it matches or exceeds full attention speed even at batch size 1 for long contexts, delivers up to 2.8$\times$ higher throughput within full attention's runnable range, and scales to million-token contexts where full attention runs out of memory. At million-token scale, ParisKV reduces decode latency by 17$\times$ and 44$\times$ compared to MagicPIG and PQCache, respectively, two state-of-the-art KV-cache Top-$k$ retrieval baselines, code is available at https://github.com/amy-77/ParisKV/tree/main.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。