动态划分键值缓存,提升长文本推理速度与精度。
DynSplit-KV: Dynamic Semantic Splitting for KVCache Compression in Efficient Long-Context LLM Inference
- 根据语义动态选择分割点,避免固定分块导致的误差。
- 相比传统方法,准确率提升49.9%,内存减少2.6倍。
- 适合需要高效处理长文本的AI系统开发者使用。
尽管键值缓存(KV Cache)对大语言模型高效推理至关重要,但其在长上下文场景下的内存占用持续增长,成为主要瓶颈,因此压缩显得尤为关键。现有压缩方法依赖固定间隔或预定义分隔符等刚性分割策略,导致不同场景下准确率下降5.5%至55.1%,根源在于语义边界具有场景依赖性。为此,我们提出动态语义分割方法DynSplit-KV:(1) 设计重要性感知的动态分隔符选择机制,将准确率提升49.9%;(2) 引入统一映射策略,将可变长度语义块转换为固定长度格式,使推理开销降低4.9倍。实验表明,该方法在长上下文场景中实现最高准确率,相较FlashAttention提升2.2倍推理速度,峰值内存减少2.6倍。
原文摘要 · Abstract (English)
Although Key-Value (KV) Cache is essential for efficient large language models (LLMs) inference, its growing memory footprint in long-context scenarios poses a significant bottleneck, making KVCache compression crucial. Current compression methods rely on rigid splitting strategies, such as fixed intervals or pre-defined delimiters. We observe that rigid splitting suffers from significant accuracy degradation (ranging from 5.5% to 55.1%) across different scenarios, owing to the scenario-dependent nature of the semantic boundaries. This highlights the necessity of dynamic semantic splitting to match semantics. To achieve this, we face two challenges. (1) Improper delimiter selection misaligns semantics with the KVCache, resulting in 28.6% accuracy loss. (2) Variable-length blocks after splitting introduce over 73.1% additional inference overhead. To address the above challenges, we propose DynSplit-KV, a KVCache compression method that dynamically identifies delimiters for splitting. We propose: (1) a dynamic importance-aware delimiter selection strategy, improving accuracy by 49.9%. (2) A uniform mapping strategy that transforms variable-length semantic blocks into a fixed-length format, reducing inference overhead by 4.9x. Experiments show that DynSplit-KV achieves the highest accuracy, 2.2x speedup compared with FlashAttention and 2.6x peak memory reduction in long-context scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。