PackInfer通过智能打包请求，显著降低大模型推理的延迟和内存开销。
PackInfer: Compute- and I/O-Efficient Attention for Batched LLM Inference
- 将不同长度请求分组打包，统一执行以平衡计算与显存访问。
- 在真实负载下降低13.0%-20.1%延迟，吞吐提升20%。
- 适合高并发、长尾请求分布的生产级大模型服务场景。
注意力效率对大语言模型推理至关重要。尽管先前工作（如FlashAttention）优化了单个请求的注意力执行，但生产环境中的大模型服务依赖批处理具有高度异构序列长度的请求以实现高吞吐量。这种不匹配导致严重的计算与I/O不平衡，加剧慢查询问题，并未充分利用GPU资源。我们提出PackInfer，一种面向异构批处理推理的核级注意力框架，支持计算与I/O感知的执行。PackInfer将批处理请求组织为负载均衡的执行组，通过将多个请求打包至统一核函数调用，有效饱和GPU利用率。通过直接在打包的查询-键区域构建注意力核，避免冗余计算并平衡线程块执行。同时引入I/O感知分组策略，将共享前缀的请求共置，并将KV缓存重新组织为组内连续布局，减少内存碎片与生成过程中的冗余数据移动。在真实工作负载上的评估显示，PackInfer相比当前最优的FlashAttention，推理延迟降低13.0%-20.1%，吞吐量提升20%。
原文摘要 · Abstract (English)
Attention efficiency is critical to large language model (LLM) inference. While prior advances optimize attention execution for individual requests (e.g., FlashAttention), production LLM serving relies on batching requests with highly heterogeneous sequence lengths for high serving throughput. This mismatch induces severe computation and I/O imbalance, exacerbates stragglers, and underutilizes GPU resources. We present PackInfer, a kernel-level attention framework that enables compute- and I/O-aware execution for heterogeneous batched inference. PackInfer orchestrates batched requests into load-balanced execution groups, effectively saturating GPU utilization by packing multiple requests into unified kernel launches. By constructing attention kernels directly over packed query-key regions, PackInfer eliminates redundant computation and balances thread-block execution. It then incorporates I/O-aware grouping that co-locates shared-prefix requests and reorganizes KV caches into group-contiguous layouts, reducing memory fragmentation and redundant data movement as generation evolves. Evaluations on real-world workloads show that PackInfer reduces inference latency by 13.0-20.1%, and improves throughput by 20% compared to the state-of-the-art FlashAttention.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。