改进低精度格式，让大模型推理更快更省资源
MX+: Pushing the Limits of Microscaling Formats for Efficient Large Language Model Serving
- 提出MX+，用扩展尾数提升异常值精度
- 4比特下性能远超现有格式，几乎无额外开销
- 适合需要高效推理的部署场景，无需改动框架
降低精度的数据格式对大语言模型（LLM）的低成本服务至关重要。尽管已有多种低精度格式，但多数需侵入式修改软件框架，或因不通用难以跨硬件厂商推广。本文聚焦近期工业界驱动的块浮点（BFP）变体，系统分析其在高效LLM服务中的极限。结果表明，现有超低比特BFP格式因块内异常值导致语言模型性能下降。为解决该问题，我们提出MX+，一种低成本、非侵入式扩展，可无缝集成至微缩放（MX）格式中。核心思想是：异常值无需使用其指数字段，可将其重用于扩展尾数，从而提升异常值精度。评估显示，MX+在4比特下显著优于现有MXFP4格式，存储开销和延迟几乎不变，是MXFP4或MXFP6的有力替代方案。
原文摘要 · Abstract (English)
Reduced-precision data formats are crucial for cost-effective serving of large language models (LLMs). While numerous reduced-precision formats have been introduced thus far, they often require intrusive modifications to the software frameworks or are rather unconventional for widespread adoption across hardware vendors. In this paper, we instead focus on recent industry-driven variants of block floating-point (BFP) formats and conduct a comprehensive analysis to push their limits for efficient LLM serving. Our analysis shows that existing ultra low-bit BFP variants struggle to provide reasonable language model performance due to outlier values in blocks. To address the outliers with BFPs, we propose MX+, a cost-effective and non-intrusive extension designed for seamless integration into the microscaling (MX) formats. MX+ builds on the key insight that the outlier does not need to use its exponent field in the element data type, which allows us to repurpose the exponent field as an extended mantissa to increase the precision of the outlier element. Our evaluation shows that MX+ achieves significantly higher model performance compared to the 4-bit MX format (MXFP4) with negligible storage overhead and slowdown, thus offering a compelling alternative to MXFP4 or MXFP6 for efficient LLM inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。