arXiv:2605.07719cs.LGcs.AI2026-05被引 1

提升长文本推理速度,让CPU和GPU协同高效处理稀疏注意力。

An Efficient Hybrid Sparse Attention with CPU-GPU Parallelism for Long-Context Inference

论文配图:An Efficient Hybrid Sparse Attention with CPU-GPU Parallelism for Long-Context Inference
图 1 · 摘自论文原文
  • 通过头属性预测与粒度预算选择,动态分配计算资源
  • 在40个任务中平均提速1.5至3.7倍,性能下降仅0.26%
  • 适合需要长上下文推理的模型部署场景

长上下文推理越来越多地依赖驻留于CPU的键值(KV)缓存,原因在于解码时的KV状态超出GPU显存容量,或在分离式预填充-解码系统中将KV数据置于主机内存。尽管块稀疏注意力在此类设置下降低了注意力计算开销,但单纯依赖稀疏性仍不足以实现端到端高效。纯GPU设计受限于PCIe带宽和元数据内存开销,而CPU-GPU混合设计仍存在显著的GPU空闲时间,以及CPU侧top-k选择和稀疏注意力计算的瓶颈。Fluxion基于三大关键洞察:输出感知的KV预算、头特异性和粒度感知的稀疏配置、跨设备协同执行。在此基础上,Fluxion结合轻量级头属性预测器、粒度预算选择器与基于优先级的调度器,联合优化预算分配、稀疏配置与CPU-GPU执行重叠。该协同设计使混合稀疏注意力在长上下文推理中同时兼顾精度与系统效率。在2个模型、3个基准、40个任务上,其性能最差仅相对完整(FULL)下降0.26%,相比最强的固定稀疏混合基线(其KV预算仅为0.05)提速1.5×–3.7×。

原文摘要 · Abstract (English)

Long-context inference increasingly operates over CPU-resident KV caches, either because decoding-time KV states exceed GPU memory capacity or because disaggregated prefill-decode systems place KV data in host memory. Although block-sparse attention reduces attention cost in this setting, sparsity alone is insufficient for end-to-end efficiency. GPU-only designs remain constrained by PCIe bandwidth and metadata memory overhead, while CPU-GPU hybrid designs still suffer from substantial GPU idle time and bottlenecks in CPU-side top-k selection and sparse attention computation. Fluxion is built on three key insights: output-aware KV budgeting, head-specific and granularity-aware sparse configuration, and cross-device coordinated execution for sparse attention over CPU-resident KV caches. Guided by these insights, Fluxion combines a lightweight head-property predictor, a granularity-budget selector, and a priority-based scheduler to jointly optimize budget allocation, sparse configuration, and CPU-GPU execution overlap. This co-design enables hybrid sparse attention to achieve both accuracy and system efficiency in long-context inference. Across 2 models, 3 benchmarks, and 40 tasks, Fluxion preserves quality well -- the worst average degradation is only -0.26 relative to FULL, while delivering 1.5$\times$-3.7$\times$ speedup over the strongest fixed sparse hybrid baseline, whose KV budget is only 0.05.

稀疏注意力长上下文混合计算推理优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。