利用2:4稀疏性加速大模型训练与推理，无精度损失。
Accelerating Transformer Inference and Training with 2:4 Activation Sparsity
- 利用平方ReLU激活的固有稀疏性实现2:4稀疏化。
- 前向和反向传播中前馈网络速度提升最高1.3倍。
- 适合追求高效推理与训练的大模型应用开发者。
本文展示如何利用2:4稀疏性——一种主流GPU硬件加速的稀疏模式——作用于激活值，以加速大语言模型的训练与推理。关键在于利用平方ReLU激活中固有的稀疏性，在不损失精度的前提下实现加速。该方法在前向与反向传播中，将前馈网络（FFN）的计算速度最高提升1.3倍。本工作凸显了稀疏性在加速大语言模型训练与推理中的巨大潜力。
原文摘要 · Abstract (English)
In this paper, we demonstrate how to leverage 2:4 sparsity, a popular hardware-accelerated GPU sparsity pattern, to activations to accelerate large language model training and inference. Crucially we exploit the intrinsic sparsity found in Squared-ReLU activations to provide this acceleration with no accuracy loss. Our approach achieves up to 1.3x faster Feed Forward Network (FFNs) in both the forwards and backwards pass. This work highlights the potential for sparsity to play a key role in accelerating large language model training and inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。