CurveFP通过共设计数值表示与乘法运算，实现低精度语言模型高效训练与推理。
CurveFP: Co-Designing Numerical Representation and Product Arithmetic for Language Models
- 采用分块对数曲线分布数值，使非零乘积可精确计算。
- CurveFP7在7B-9B模型上比FP8低1位元仍保持接近原生精度。
- 适合追求高能效的低精度推理系统，尤其适合硬件加速器部署。
低精度格式通常优化标量保真度，但沿用传统乘法运算。我们提出CurveFP，一种块缩放族，将数值分布在交错的对数曲线上。统一的曲线索引使每个非零乘积变为符号与整数索引的精确更新，有理基数揭示了累加所需的有限相位调度。我们实例化出用于训练的CurveFP8 E4C3/E5C2和用于紧凑推理的CurveFP7 E3C3。在四个7B-9B模型上，CurveFP7在少一位元素位的情况下优于逐张量FP8困惑度，且保持在1.32%以内原生质量。CurveFP8在所有36组训练-GEMM对比中降低误差。在三个匹配的3B-token预训练三元组中，其平均BF16推理困惑度为22.5366，优于FP8的22.5407，且每种子实验均表现更优。下游评估显示迁移性能一致，并在WikiText-103上持续提升。在初步4x4 Nangate45空间加速器单元中，CurveFP8在500MHz下比时序关键的FP8少用一个乘积寄存器，面积减少4.6%。这些结果支持CurveFP作为数值与算术的协同设计，系统级效率留待未来研究。
原文摘要 · Abstract (English)
Low-precision formats usually optimize scalar fidelity while inheriting conventional product arithmetic. We introduce CurveFP, a block-scaled family that distributes magnitudes across interleaved logarithmic curves. Uniform curve indices make every nonzero product an exact sign and integer-index update, while a rational radix exposes the finite phase schedule required for accumulation. We instantiate the algebra as CurveFP8 E4C3/E5C2 for training and CurveFP7 E3C3 for compact inference. On four 7B-9B models, CurveFP7 beats tensorwise FP8 perplexity with one fewer element bit and stays within 1.32% of native quality. CurveFP8 lowers error in all 36 paired training-GEMM comparisons. Across three matched 3B-token pretraining triplets, it reaches mean BF16-inference perplexity 22.5366 versus 22.5407 for FP8 and has a lower format penalty in every seed. Downstream evaluation shows transfer parity and a consistent WikiText-103 gain. In a preliminary 4x4 Nangate45 spatial accelerator tile, CurveFP8 uses one fewer product register and 4.6% less area than timing-closing FP8 at 500 MHz. These results support CurveFP as a numerical and arithmetic co-design, while leaving system-level efficiency to future study.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。