用知识蒸馏恢复量化后大模型精度，稳定高效。
Quantization-Aware Distillation for NVFP4 Inference Accuracy Recovery
- 在量化模型上用KL散度蒸馏全精度教师模型，提升精度。
- 多阶段训练模型量化后精度可恢复至接近BF16水平。
- 不依赖完整数据集，适合工程复杂场景的模型优化。
本技术报告提出量化感知蒸馏（QAD），用于恢复NVFP4量化大型语言模型（LLMs）和视觉语言模型（VLMs）的精度。QAD通过KL散度损失，将全精度教师模型的知识蒸馏到量化学生模型中。尽管对量化模型进行蒸馏并非新方法，但QAD在当前主流大模型中展现出显著优势：1. 在经过多阶段后训练流程（包括监督微调、强化学习和模型合并）的模型上表现优异，相比传统量化感知训练（QAT）更稳定且工程实现更简单；2. 对数据质量与覆盖范围不敏感，无需完整训练数据即可实现精度恢复。我们在AceReason Nemotron、Nemotron 3 Nano、Nemotron Nano V2、Nemotron Nano V2 VL（VLM）及Llama Nemotron Super v1等多个后训练模型上验证了QAD，均实现精度一致恢复至接近BF16水平。
原文摘要 · Abstract (English)
This technical report presents quantization-aware distillation (QAD) and our best practices for recovering accuracy of NVFP4-quantized large language models (LLMs) and vision-language models (VLMs). QAD distills a full-precision teacher model into a quantized student model using a KL divergence loss. While applying distillation to quantized models is not a new idea, we observe key advantages of QAD for today's LLMs: 1. It shows remarkable effectiveness and stability for models trained through multi-stage post-training pipelines, including supervised fine-tuning (SFT), reinforcement learning (RL), and model merging, where traditional quantization-aware training (QAT) suffers from engineering complexity and training instability; 2. It is robust to data quality and coverage, enabling accuracy recovery without full training data. We evaluate QAD across multiple post-trained models including AceReason Nemotron, Nemotron 3 Nano, Nemotron Nano V2, Nemotron Nano V2 VL (VLM), and Llama Nemotron Super v1, showing consistent recovery to near-BF16 accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。