arXiv:2608.20953cs.CLcs.AI2026-08

4bit压缩大模型推理性能差?用新方法快速恢复,效果更好且省显存。

Quantization-Aware Healing: A Practical Recipe for Recovering Compressed, 4-Bit LLMs

  • 从原始未压缩模型直接蒸馏4比特学生模型,跳过传统微调路径。
  • 在9个基准中7个超越或持平原版,参数减半、显存降至1/4。
  • 训练快7倍且稳定,无需调参,适合工业级快速部署。

低成本部署大语言模型越来越依赖结构压缩和4比特量化,但这两步会严重损害推理、数学、编程和长上下文能力,需恢复(即“修复”)阶段才能使用。传统方案量化感知训练(QAT)对压缩后的4比特模型重新拟合硬标签,但在我们的流程中收敛慢且性能崩溃。我们改用量化感知修复(QAH),因结构压缩模型从未以全精度独立训练,其bfloat16检查点仅为原始模型的近似蒸馏结果;因此QAH直接从原始未压缩模型蒸馏4比特学生模型。在从GPT-OSS 120B到60B再到MXFP4的流水线中,该方法生成的学生模型在9个基准中有7个达到或超过其bfloat16源模型表现,权重内存约为1/4,参数量为教师的一半,已开源为Hypernova-60B。相比匹配的QAT基线,其达到相当峰值速度提升约7倍,且持续训练中保持稳定,无需手动早停。此外报告了部署经验,包括分布式训练后端间显著可复现的质量差异。目标是提供一种无需数周超参搜索即可部署的实用方案。

原文摘要 · Abstract (English)

Serving large language models cheaply increasingly means shipping models that are both structurally compressed to a fraction of their parameters and quantized to 4 bits. Together these steps degrade reasoning, mathematics, coding, and long-context behavior enough to require a recovery, or healing, stage before deployment. The default recipe, quantization-aware training (QAT), re-fits the compressed, quantized model to hard labels; in our pipeline it converged slowly and collapsed past its peak. We adopted Quantization-Aware Healing (QAH) instead. Because a structurally compressed model is never independently trained at full precision, its bfloat16 checkpoint is a distillation-recovered approximation of the original; QAH distills the 4-bit student directly from the original, uncompressed model. On a GPT-OSS 120B to 60B to MXFP4 pipeline, the QAH student matches or beats its bfloat16 source on 7 of 9 benchmarks at roughly 4 times less weight memory and half the teacher's parameter count, and is released open-weight as Hypernova-60B. Against a matched QAT baseline it reaches a comparable peak about 7 times faster and stays stable under continued training, without hand-tuned early stopping. We also report deployment lessons, including a large, reproducible quality gap between distributed-training backends. Our aim is a recipe deployable without a multi-week hyper-parameter search.

模型压缩4比特量化蒸馏大模型部署

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。