动态块级回退让INT8训练更稳定，提速1.57倍
Accurate INT8 Training Through Dynamic Block-Level Fallback
- 对含异常激活的块动态切换至16位计算
- 在RTX4090上实现1.57倍端到端训练加速
- 适合需高效训练GLU结构模型的研究者
Transformer模型在各类AI应用中取得显著成功，但训练成本高昂。低比特训练（如INT8）可利用更高吞吐的计算单元，已在GPT2模型上通过块级量化证明有效。然而，对于包含GLU单元的现代Transformer变体，其激活值分布复杂，导致训练困难。为此，我们提出回退量化，采用混合精度矩阵乘法，在包含异常值的激活块上动态从8位回落至16位。实验表明，该方法在微调与预训练场景下均表现稳健。此外，在RTX4090 GPU上实现了1.57倍的端到端训练速度提升。
原文摘要 · Abstract (English)
Transformer models have achieved remarkable success across various AI applications but face significant training costs. Low-bit training, such as INT8 training, can leverage computational units with higher throughput, and has already demonstrated its effectiveness on GPT2 models with block-level quantization. However, it struggles with modern Transformer variants incorporating GLU units. This is because those variants demonstrate complex distributions of activation outliers. To address the challenge, we propose Fallback Quantization, implementing mixed-precision GEMM that dynamically falls back 8-bit to 16-bit for activation blocks containing outliers. Experiments show that our approach is robustly competent in both fine-tuning and pretraining settings. Moreover, our method achieves a 1.57x end-to-end training speedup on RTX4090 GPUs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。