多智能体共用压缩KV缓存，内存减少97.7%仍保持高精度。
PolyKV: A Shared Asymmetrically-Compressed KV Cache Pool for Multi-Agent LLM Inference
- 用不对称压缩共享单个KV缓存池，键用int8量化，值用FWHT+3bit量化。
- 15个智能体共享4K上下文时，缓存从19.8GB降至0.45GB，压缩比2.91x。
- 支持多读并发，困惑度仅增0.57%，适合大规模多智能体推理场景。
我们提出PolyKV，一种多个并发推理智能体共享单一非对称压缩KV缓存池的系统。不同于传统为每个智能体单独分配KV缓存的方式，PolyKV仅写入一次压缩缓存，并通过HuggingFace DynamicCache对象注入到N个独立智能体上下文中。压缩采用非对称策略：键使用int8（q8_0）量化以保持softmax稳定性，值则采用TurboQuant MSE——基于快速沃尔什-哈达玛变换（FWHT）旋转后，再进行3位Lloyd-Max量化，中心点针对N(0,1)分布优化。我们在两个模型规模（SmolLM2-1.7B-Instruct和Llama-3-8B-Instruct）、三种上下文长度（600-7,194令牌）及最多15个并发智能体下进行了评估。PolyKV在所有配置下均实现稳定的2.91倍压缩率。在Llama-3-8B模型上，15个智能体共享4K令牌上下文时，KV缓存内存由19.8 GB降至0.45 GB，减少97.7%，同时仅带来+0.57%的困惑度退化，平均BERTScore F1为0.928。困惑度差值不随智能体数量增加而上升，且随上下文长度增长反而下降至-0.26%（1,851个连贯令牌）。据我们所知，此前无工作将单一共享、有损压缩的KV池与多读并发智能体访问相结合。
原文摘要 · Abstract (English)
We present PolyKV, a system in which multiple concurrent inference agents share a single, asymmetrically compressed KV cache pool. Rather than allocating a separate KV cache per agent -- the standard paradigm -- PolyKV writes a compressed cache once and injects it into N independent agent contexts via HuggingFace DynamicCache objects. Compression is asymmetric: Keys are quantized at int8 (q8_0) to preserve softmax stability, while Values are compressed using TurboQuant MSE -- a Fast Walsh-Hadamard Transform (FWHT) rotation followed by 3-bit Lloyd-Max quantization with centroids tuned to N(0,1). We evaluate across two model scales (SmolLM2-1.7B-Instruct and Llama-3-8B-Instruct), three context lengths (600-7,194 tokens), and up to 15 concurrent agents. PolyKV achieves a stable 2.91x compression ratio across all configurations. On Llama-3-8B with 15 agents sharing a 4K-token context, PolyKV reduces KV cache memory from 19.8 GB to 0.45 GB -- a 97.7% reduction -- while maintaining only +0.57% perplexity degradation and a mean BERTScore F1 of 0.928. PPL delta does not grow with agent count and improves as context length increases, inverting to -0.26% at 1,851 coherent tokens. To our knowledge, no prior work combines a single shared, lossy-compressed KV pool with multi-reader concurrent agent access.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。