混合CPU-GPU注意力让大模型长文本生成更快更准
HGCA: Hybrid GPU-CPU Attention for Long Context LLM Inference
- GPU存近期数据做密集计算，CPU存关键数据做稀疏计算
- 支持更长序列和更大批量，速度比现有方法快且准确率更高
- 无需重训练，直接接入现有框架，适合部署长文本应用
大语言模型推理受制于有限的GPU显存，尤其在长上下文生成时需存储大量键值（KV）缓存。现有方法或把KV缓存移至CPU内存，或采用稀疏注意力减少GPU负载，但常低估CPU算力并影响精度。本文提出HGCA，一种混合式CPU-GPU注意力机制，实现高吞吐、可扩展的推理，几乎保持全注意力质量。HGCA在GPU中对最近生成的KV条目执行密集注意力，在CPU内存中对精选的重要KV条目并行执行稀疏注意力，通过对数求和-合并技术高效融合输出，显著降低PCIe传输开销。同时引入细粒度、每头优化的稀疏化策略，适配CPU执行，保留语义相关性的同时降低计算量。实现无缝集成至现有LLM框架，无需模型重训练。跨多种模型与工作负载的实验表明，HGCA在扩展性、支持更长序列和更大批处理方面表现优异，性能与精度均优于现有稀疏注意力基线，所有测试均在普通GPU硬件上完成。
原文摘要 · Abstract (English)
Scaling inference for large language models (LLMs) is increasingly constrained by limited GPU memory, especially due to growing key-value (KV) caches required for long-context generation. While existing approaches offload KV caches to CPU memory or apply sparse attention to reduce GPU load, they often underutilize CPU compute resources and compromise accuracy. We present HGCA, a hybrid CPU-GPU attention mechanism that enables scalable, high-throughput LLM inference with near-full attention quality. HGCA performs dense attention on recently generated KV entries retained in GPU memory and parallel sparse attention on selected, salient KV entries in CPU memory. The attention outputs are efficiently merged using log-sum-exp fusion, minimizing PCIe transfer overhead. HGCA also introduces a finegrained, per-head sparsification strategy optimized for CPU execution, preserving contextual relevance while reducing computation. Our implementation seamlessly integrates into existing LLM frameworks without requiring model retraining. Experiments across diverse models and workloads show that HGCA achieves superior scalability, supports longer sequences and larger batch sizes, and outperforms existing sparse attention baselines in both performance and accuracy -- all on commodity GPU hardware.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。