提出新方法解决低精度训练中的隐性失效问题，让模型更稳定可靠。
Max-Window Scale Estimation for Near-Lossless HiF8 W8A8 Quantization-Aware Training
- 用64步历史窗口的保守最大值算法缓解量化饱和问题。
- 500步双精度预热+10^-5学习率，减少知识遗忘，提升性能。
- 适合追求高效部署的LLM研究者与工程师参考。
低比特浮点格式的量化感知训练（QAT）可实现大语言模型的高效部署，但会引入标准训练指标无法察觉的细微失效。本文通过延迟张量缩放（DTS）视角，对OpenPangu-Embedded-1B进行八组受控实验，识别并分离出两种独立的失效模式：(i) amax饱和，即延迟的缩放估计在前向传播中导致敏感表示被截断；(ii) 灾难性遗忘，即激进的学习率覆盖了预训练的常识知识。二者均无法从训练损失中检测。我们通过在64步历史窗口上采用保守的最大值算法解决amax饱和，并结合500步BF16预热后以10^{-5}学习率进行QAT缓解遗忘。二者协同作用，最终配置在10,000步训练中仅产生0.11%训练损失平均绝对误差，相较匹配的BF16基线，实现0.43% MMLU下降、0.58% HellaSwag下降和0.22% ARC-Challenge下降。
原文摘要 · Abstract (English)
Quantization-aware training (QAT) with low-bit floating-point formats enables efficient LLM deployment, yet introduces subtle failure modes invisible to standard training metrics. We present a systematic study of HiF8 W8A8 QAT for OpenPangu-Embedded-1B through the lens of Delayed Tensor Scaling (DTS). Across eight controlled experiments, we identify and disentangle two orthogonal failure modes: (i)amax saturation, where delayed scale estimates silently corrupt knowledge-sensitive representations via forward-pass clipping, and (ii)catastrophic forgetting, where an aggressive learning rate overwrites pretrained commonsense knowledge independently of quantization. Neither is detectable from training loss alone. We address amax saturation with a conservative max-algorithm DTS strategy over a 64-step history window, and mitigate forgetting via a 500-step BF16 warmup followed by QAT at lr=10^{-5}. Both fixes are necessary and sufficient: our final configuration achieves 0.43% MMLU drop, 0.58% HellaSwag drop, and 0.22% ARC-Challenge drop versus a matched BF16 baseline, with a training loss APE of only 0.11% over 10,000 steps.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。