提出新型8位浮点格式HiFloat8，兼顾精度与动态范围，支持训练推理全流程。
Ascend HiFloat8 Format for Deep Learning
- 采用分层精度设计：7/8/16个指数对应3/2/1位尾数，平衡精度与范围。
- 扩展非规格化范围至38个二进制段（原FP16为40），支持更广动态范围。
- 可同时用于前向与反向传播，适合大模型训练和推理部署。
本文提出一种面向深度学习的新型8位浮点数据格式HiFloat8（简称HiF8）。HiF8采用分层精度设计：常规值编码包含7个指数配3位尾数、8个指数配2位尾数、16个指数配1位尾数；非规格化值编码通过额外7个2的幂扩展动态范围，达到38个二进制段（binades），接近FP16的40个。除正零与负零共用一个比特模式外，其余特殊值均被编码。得益于精度与动态范围的更好平衡，HiF8可同时用于AI训练的前向与反向传播。本文还介绍了HiF8的定义、舍入方法及初步的训练与推理方案，并通过多种神经网络（包括传统网络与大语言模型）的大规模仿真验证其有效性。
原文摘要 · Abstract (English)
This preliminary white paper proposes a novel 8-bit floating-point data format HiFloat8 (abbreviated as HiF8) for deep learning. HiF8 features tapered precision. For normal value encoding, it provides 7 exponent values with 3-bit mantissa, 8 exponent values with 2-bit mantissa, and 16 exponent values with 1-bit mantissa. For denormal value encoding, it extends the dynamic range by 7 extra powers of 2, from 31 to 38 binades (notice that FP16 covers 40 binades). Meanwhile, HiF8 encodes all the special values except that positive zero and negative zero are represented by only one bit-pattern. Thanks to the better balance between precision and dynamic range, HiF8 can be simultaneously used in both forward and backward passes of AI training. In this paper, we will describe the definition and rounding methods of HiF8, as well as the tentative training and inference solutions. To demonstrate the efficacy of HiF8, massive simulation results on various neural networks, including traditional neural networks and large language models (LLMs), will also be presented.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。