针对4比特训练中异常值问题，提出动态补偿方法显著缩小与高精度训练差距。
Dissecting Outlier Dynamics in LLM NVFP4 Pretraining
- 识别出注意力和前馈层中导致异常值的关键组件及其演化规律。
- 通过在线补偿机制将损失差距从0.94%降至0.58%，保持下游性能。
- 适合关注低比特大模型训练效率与精度平衡的研究者。
使用4比特算术训练大语言模型可提升吞吐量与内存效率，但其有限动态范围使模型对异常值更敏感。尽管NVFP4通过分层微缩放缓解量化误差，仍存在与BF16的持续损失差距。本研究对NVFP4预训练过程中的异常值动态进行纵向分析，聚焦其定位、成因及时间演化。结果表明：相较于Softmax Attention（SA），Linear Attention（LA）虽降低每张量尾部极端值，但在块量化下仍出现显著块级峰值；异常值主要源自SA中的Softmax、LA中的gating以及FFN中的SwiGLU，且“后QK”操作对量化更敏感。值得注意的是，异常值由训练初期的瞬时尖峰演变为后期少数持续高温通道。基于此，提出Hot-Channel Patch（HCP）在线补偿机制，利用硬件高效核识别高温通道并重注入残差。进一步构建CHON训练方案，集成HCP与后QK保护。在1.3B参数的GLA模型上，经600亿词元训练，该方案将损失差距从0.94%降至0.58%，同时维持下游任务准确率。
原文摘要 · Abstract (English)
Training large language models using 4-bit arithmetic enhances throughput and memory efficiency. Yet, the limited dynamic range of FP4 increases sensitivity to outliers. While NVFP4 mitigates quantization error via hierarchical microscaling, a persistent loss gap remains compared to BF16. This study conducts a longitudinal analysis of outlier dynamics across architecture during NVFP4 pretraining, focusing on where they localize, why they occur, and how they evolve temporally. We find that, compared with Softmax Attention (SA), Linear Attention (LA) reduces per-tensor heavy tails but still exhibits persistent block-level spikes under block quantization. Our analysis attributes outliers to specific architectural components: Softmax in SA, gating in LA, and SwiGLU in FFN, with "post-QK" operations exhibiting higher sensitivity to quantization. Notably, outliers evolve from transient spikes early in training to a small set of persistent hot channels (i.e., channels with persistently large magnitudes) in later stages. Based on these findings, we introduce Hot-Channel Patch (HCP), an online compensation mechanism that identifies hot channels and reinjects residuals using hardware-efficient kernels. We then develop CHON, an NVFP4 training recipe integrating HCP with post-QK operation protection. On GLA-1.3B model trained for 60B tokens, CHON reduces the loss gap to BF16 from 0.94% to 0.58% while maintaining downstream accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。