优化4位浮点量化模型推理速度，提升显存效率。
Fast NF4 Dequantization Kernels for Large Language Model Inference
- 利用共享内存优化反量化流程，降低计算开销。
- 在三个大模型上实现2.0~2.2倍内核加速，端到端提速1.54倍。
- 仅需64字节共享内存，适配HuggingFace生态，即插即用。
大语言模型（LLMs）规模已超出单块GPU显存容量，需依赖量化技术部署。尽管NF4（4-bit NormalFloat）量化可实现4倍内存压缩，但在当前NVIDIA GPU（如Ampere A100）上进行推理时，仍需昂贵的反量化至FP16，成为性能瓶颈。本文提出一种轻量级共享内存优化方案，通过合理利用内存层级结构，在保持完整生态兼容性的前提下解决该问题。与开源BitsAndBytes实现相比，本方法在Gemma 27B、Qwen3 32B和Llama3.3 70B三个模型上取得2.0–2.2倍内核加速，端到端性能提升最高达1.54倍，得益于共享内存相较全局内存12–15倍的延迟优势。优化通过简化索引逻辑减少指令数，每线程块仅使用64字节共享内存，证明轻量级改进亦能带来显著性能增益。该工作为HuggingFace生态系统提供即插即用方案，使现有硬件更易运行先进大模型。
原文摘要 · Abstract (English)
Large language models (LLMs) have grown beyond the memory capacity of single GPU devices, necessitating quantization techniques for practical deployment. While NF4 (4-bit NormalFloat) quantization enables 4$\times$ memory reduction, inference on current NVIDIA GPUs (e.g., Ampere A100) requires expensive dequantization back to FP16 format, creating a critical performance bottleneck. This paper presents a lightweight shared memory optimization that addresses this gap through principled memory hierarchy exploitation while maintaining full ecosystem compatibility. We compare our technique against the open-source BitsAndBytes implementation, achieving 2.0--2.2$\times$ kernel speedup across three models (Gemma 27B, Qwen3 32B, and Llama3.3 70B) and up to 1.54$\times$ end-to-end improvement by leveraging the 12--15$\times$ latency advantage of shared memory over global memory access. Our optimization reduces instruction counts through simplified indexing logic while using only 64 bytes of shared memory per thread block, demonstrating that lightweight optimizations can deliver substantial performance gains with minimal engineering effort. This work provides a plug-and-play solution for the HuggingFace ecosystem that democratizes access to advanced models on existing GPU infrastructure.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。