通过异构计算压缩大模型注意力缓存，实现极致内存节省。
HCAttention: Extreme KV Cache Compression via Heterogeneous Attention Computing for LLMs
- 融合量化、值卸载与动态淘汰，实现跨GPU-CPU协同推理。
- 在仅保留12.5%缓存时仍保持性能，400万token输入无损。
- 无需微调即可部署，适合长文本生成与资源受限场景。
大语言模型处理长上下文输入时，键值（KV）缓存占用巨大内存，成为瓶颈。现有压缩方法在内存压缩超85%时性能明显下降，且基于GPU-CPU协作的近似注意力策略尚未充分探索。本文提出HCAttention，一种异构注意力计算框架，集成关键量化、值卸载与动态KV淘汰，可在极端内存约束下实现高效推理。该方法兼容现有Transformer架构，无需模型微调。LongBench基准测试显示，其将KV缓存缩减至原始大小的25%，仍保持全注意力模型精度；尤为显著的是，在仅保留12.5%缓存时仍具竞争力，创下大模型KV缓存压缩新纪录。据我们所知，HCAttention首次使Llama-3-8B模型在单张80GB显存A100 GPU上处理400万词元输入。
原文摘要 · Abstract (English)
Processing long-context inputs with large language models presents a significant challenge due to the enormous memory requirements of the Key-Value (KV) cache during inference. Existing KV cache compression methods exhibit noticeable performance degradation when memory is reduced by more than 85%. Additionally, strategies that leverage GPU-CPU collaboration for approximate attention remain underexplored in this setting. We propose HCAttention, a heterogeneous attention computation framework that integrates key quantization, value offloading, and dynamic KV eviction to enable efficient inference under extreme memory constraints. The method is compatible with existing transformer architectures and does not require model fine-tuning. Experimental results on the LongBench benchmark demonstrate that our approach preserves the accuracy of full-attention model while shrinking the KV cache memory footprint to 25% of its original size. Remarkably, it stays competitive with only 12.5% of the cache, setting a new state-of-the-art in LLM KV cache compression. To the best of our knowledge, HCAttention is the first to extend the Llama-3-8B model to process 4 million tokens on a single A100 GPU with 80GB memory.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。