针对昇腾芯片优化低比特推理，提出新型浮点格式提升大模型效率。
Unleashing Low-Bit Inference on Ascend NPUs: A Comprehensive Evaluation of HiFloat Formats
- 设计专为昇腾NPU优化的分层浮点格式HiFloat（HiF8/HiF4）
- 4比特下避免精度崩溃，比整数格式更稳定可靠
- 兼容主流后训练量化框架，适合部署在昇腾设备上
随着大语言模型规模扩大，低比特浮点格式如MXFP和NVFP4为精度与效率提供了新可能。本文评估了专为昇腾NPU设计的HiFloat（HiF8和HiF4）系列格式。通过在权重-激活和键值缓存任务中的严格对比，得出三个关键结论：(1) INT8适用于范围窄的数据，而浮点格式在高方差数据上表现更优；(2) 在4比特场景下，HiF4的分层缩放机制有效防止了整数格式常见的精度坍塌；(3) HiFloat完全兼容当前最先进的后训练量化框架。总体而言，HiFloat为在NPU上实现高效的大模型推理提供了可行方案。
原文摘要 · Abstract (English)
As LLMs scale, low-bit floating-point formats like MXFP and NVFP4 offer new opportunities for precision and efficiency. In this work, we evaluate HiFloat (HiF8 and HiF4), a family of formats tailored for Ascend NPUs. Through rigorous comparison across weight-activation and KV-cache tasks, we provide three key insights: (1) INT8 suits narrow-range data, while floating-point formats excel with high-variance data; (2) in 4-bit regimes, HiF4's hierarchical scaling prevents the accuracy collapse seen in integer formats; and (3) HiFloat is fully compatible with state-of-the-art post-training quantization frameworks. Overall, HiFloat provides a solution for high-efficiency LLM inference on NPUs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。