提出新方法消除大模型训练中的极端异常值，实现无需复杂工程的FP8训练
TWEO: Transformers Without Extreme Outliers Enables FP8 Training And Quantization For Dummies
- 通过简单损失函数抑制权重矩阵共线性导致的异常值
- 将极端异常值从10000+降至20以下，支持全模型FP8训练
- 适合希望低成本落地大模型量化与高效训练的研究者
现代硬件原生支持FP8对大模型训练至关重要，但严重受制于极端激活异常值。现有方案依赖复杂的混合精度设计或侵入式结构修改。本文挑战了异常值源于数据的传统认知，证明其是训练中由权重矩阵特定结构（如共线性）机械产生的数据无关产物。基于此，提出非侵入式新损失函数TWEO，仅通过简单损失项即可将异常值从10000+降至20以下。该方法使LLM和ViT均能实现无需工程技巧或架构改动的全模型FP8预训练。标准FP8训练崩溃时，TWEO性能接近BF16基线，且训练吞吐提升36%。同时，首次实现硬件友好的W8A8每张量静态量化在大模型上的表现达到当前最优。
原文摘要 · Abstract (English)
Native FP8 support in modern hardware is essential for training large Transformers, but is severely hindered by extreme activation outliers. Existing solutions either rely on complex mixed-precision engineering or invasive architectural modifications. This paper fundamentally challenges the conventional wisdom that outliers are data-driven. We demonstrate that extreme outliers are a data-independent, mechanically-produced artifact of training, originating from specific structural properties of the weight matrices (i.e., colinearity). Based on this insight, we propose TWEO (Transformers Without Extreme Outliers), a novel, non-invasive loss function. TWEO effectively prevents extreme outliers via a very simple loss term, which reduces outliers from 10000+ to less than 20. TWEO then enables full-model FP8 pre-training with neither engineering tricks nor architectural changes for both LLM and ViT. When standard FP8 training catastrophically collapses, TWEO achieves performance comparable to the BF16 baseline while delivering a 36% increase in training throughput. Also, TWEO enables a new quantization paradigm. Hardware-friendly W8A8 per-tensor static quantization of LLMs, previously considered completely unusable due to outliers, achieves SOTA performance for the first time on TWEO-trained models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。