通过细粒度融合优化大模型激活计算，显著提升推理速度。
Tile-Level Activation Overlap for Efficient LLM Inference
- 在计算块层面融合SwiGLU与GeMM，减少中间张量开销。
- 实测最高提速2.47倍，内存瓶颈转为计算瓶颈，达到79.5%算力利用率。
- 手写内核优于自动编译，适合高性能大模型部署场景。
现代大语言模型中，SwiGLU是主流的MLP激活函数，但其中间张量生成占用了9-37%的MLP执行时间。本文提出两种基于CUTLASS的SM90内核，从计算块层面将SwiGLU与GeMM融合。内核1采用乒乓调度，使门控累加器上的Swish计算与上行块加载重叠；内核2通过自定义结尾访问树，将SwiGLU计算与块存储交错。在NVIDIA H100上对Qwen-2.5系列（0.5B-72B）模型的评估显示，相比PyTorch，最高可实现2.47倍加速，将工作负载从内存受限转为计算受限，峰值BF16利用率达79.5%。实验表明，torch.compile无法复现此融合效果（慢3-7倍），验证了手写块级设计的必要性。此外，融合内核在数值精度上更优，零误差匹配，而cuBLAS存在4.5%-11%的误差率。
原文摘要 · Abstract (English)
SwiGLU is the dominant MLP activation in modern large language models, yet its intermediate tensor materialization costs 9-37% of MLP execution time. We present two complementary CUTLASS-based SM90 kernels that fuse SwiGLU into GeMM at the tile level. Kernel-1 overlaps Swish computation on the Gate accumulator with Up-tile loading using the Pingpong warp-specialized schedule; Kernel-2 interleaves SwiGLU with tile stores via a custom Epilogue Visitor Tree. Evaluated on Qwen-2.5 models (0.5B-72B) on NVIDIA H100, our kernels achieve up to 2.47x speedup over PyTorch, shifting workloads from memory-bound to compute-bound and reaching 79.5% peak BF16 utilization. We demonstrate that torch.compile cannot replicate this fusion (3-7x slower than our kernels), validating the need for hand-crafted tile-level design. Our fused kernels are also numerically superior, achieving zero mismatches compared to 4.5-11% for cuBLAS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。