将Transformer全过程融合为单一内核，加速低批量推理
FlashFormer: Whole-Model Kernels for Efficient Low-Batch Inference
- 把整个Transformer前向传播合并成一个内核，减少调用开销
- 在多种模型规模和量化设置下，相比现有内核实现显著提速
- 适合边缘部署和低延迟场景的高效推理需求
现代大语言模型的规模和计算特性促使人们开发针对特定训练与推理负载的专用内核。现有内核主要优化计算利用率，聚焦于大批次训练与推理场景。然而，在边缘部署和延迟敏感应用中，低批次推理仍至关重要，其性能受内存带宽和内核启动开销显著影响。本文提出FlashFormer，将整个Transformer前向传播融合为单一内核，以加速大语言模型的低批次推理。在多种模型规模和量化设置下，FlashFormer相较现有推理内核实现了非平凡的速度提升。
原文摘要 · Abstract (English)
The size and compute characteristics of modern large language models have led to an increased interest in developing specialized kernels tailored for particular training and inference workloads. Existing kernels primarily optimize for compute utilization, targeting the large-batch training and inference settings. However, low-batch inference, where memory bandwidth and kernel launch overheads are significant factors, remains important for many applications of interest such as in edge deployment and latency-sensitive applications. This paper describes FlashFormer, which fuses the entire transformer forward pass into a single kernel for accelerating low-batch inference of large language models. Across various model sizes and quantizations settings, FlashFormer achieves nontrivial speedups compared to existing inference kernels.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。