通过分层重要性感知压缩，显著降低大模型解码时的内存占用和能耗。
HiKV: Hierarchical Importance-Aware KV Cache with Hardware Acceleration for LLM Decoding
- 分两阶段压缩KV缓存：先剔除不重要词元，再只保留关键信息。
- 实测提升7.95倍推理速度，能耗降低90%，精度损失仅1%。
- 专用硬件加速器仅增加8%面积，适合部署在资源受限设备上。
随着长上下文大语言模型的广泛应用，解码过程中不断增长的KV缓存已成为主要的内存瓶颈。为此，我们提出HiKV——一种算法与硬件协同设计的新方法，通过分层重要性感知利用KV缓存冗余。算法上，HiKV在两个粒度上压缩缓存：第一阶段在固定预算内剔除不重要的词元；第二阶段仅加载每个保留词元的关键部分，实现单粒度无法达到的压缩率。架构上，设计了一个可重构的重要性排序器，可在两阶段间切换不同排序路径，以最小开销统一两阶段加速。在代表性LLM上的评估显示，相比原始缓存基线，HiKV在保持约1%精度损失的前提下，实现最高7.95倍的速度提升和90%的注意力计算能耗降低。在同精度约束下，相较现有基于重要性的方法，进一步减少1.82~4.87倍外部内存访问。这些优势由专用硬件组件实现，系统面积仅增加8%。
原文摘要 · Abstract (English)
With the rapid adoption of long-context large language models (LLMs), the continuously growing KV cache during decoding has become the critical memory bottleneck. To tackle this challenge, we propose HiKV, a novel algorithm-hardware co-design that exploits KV cache redundancy through hierarchical importance awareness. Algorithmically, HiKV compresses the KV cache at two granularities: Stage I evicts unimportant tokens within a fixed budget, and Stage II further loads only the significant elements of each retained token, reaching compression ratios unattainable at a single granularity. Architecturally, we develop a dedicated accelerator centered on a reconfigurable importance sorter that switches between the distinct sorting datapaths each stage requires, unifying the two-stage acceleration in one circuit with minimal overhead. Evaluated on representative LLMs, HiKV achieves up to 7.95x speedup and 90% energy reduction in the attention computation over the vanilla KV cache baseline within negligible 1% accuracy loss. Under iso-accuracy constraints, HiKV outperforms state-of-the-art importance-based methods by achieving an additional 1.82~4.87x reduction in external memory accesses. These benefits are enabled by specialized hardware components that add only 8% to the system area.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。