4-bit量化几乎不降性能，3比特动态量化实现高效部署。
Quantitative Analysis of Performance Drop in DeepSeek Model Quantization
- 提出动态3比特量化DQ3_K_M，提升传统方法表现
- 4比特量化在多任务中接近原始精度，支持单机部署
- 适配NVIDIA和华为芯片，开源模型供研究使用
随着本地部署需求上升，DeepSeek-R1与V3的671B参数量级在标准8卡设备上超出内存限制。本文首次对完整深度探索模型系列进行多比特量化定量评估。结果表明，4比特量化在保持近似原始性能的同时，可实现单机部署。进一步提出动态3比特量化方法DQ3_K_M，其在多项基准测试中显著优于传统Q3_K_M，并在多数任务中媲美4比特量化（Q4_K_M）。该方法支持NVIDIA H100/A100及华为910B单机部署。相关实现已开源至https://github.com/UnicomAI/DeepSeek-Eval，包含优化后的3比特版DeepSeek-R1与DeepSeek-V3。
原文摘要 · Abstract (English)
Recently, there is a high demand for deploying DeepSeek-R1 and V3 locally, possibly because the official service often suffers from being busy and some organizations have data privacy concerns. While single-machine deployment offers infrastructure simplicity, the models' 671B FP8 parameter configuration exceeds the practical memory limits of a standard 8-GPU machine. Quantization is a widely used technique that helps reduce model memory consumption. However, it is unclear what the performance of DeepSeek-R1 and V3 will be after being quantized. This technical report presents the first quantitative evaluation of multi-bitwidth quantization across the complete DeepSeek model spectrum. Key findings reveal that 4-bit quantization maintains little performance degradation versus FP8 while enabling single-machine deployment on standard NVIDIA GPU devices. We further propose DQ3_K_M, a dynamic 3-bit quantization method that significantly outperforms traditional Q3_K_M variant on various benchmarks, which is also comparable with 4-bit quantization (Q4_K_M) approach in most tasks. Moreover, DQ3_K_M supports single-machine deployment configurations for both NVIDIA H100/A100 and Huawei 910B. Our implementation of DQ3\_K\_M is released at https://github.com/UnicomAI/DeepSeek-Eval, containing optimized 3-bit quantized variants of both DeepSeek-R1 and DeepSeek-V3.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。