仅加一个RMSNorm就能让大模型稳定量化到1.58比特。
An Extra RMSNorm is All You Need for Fine Tuning to 1.58 Bits
- 在每个线性层前加RMSNorm，配合分层渐进量化策略。
- 在标准任务上达到或超过知识蒸馏效果，精度接近全精度模型。
- 适合追求极致压缩与推理效率的部署场景。
大型语言模型（LLMs）虽已变革自然语言处理，但其规模导致实际部署成本高昂。后训练量化可降低内存与计算开销，但常伴随精度下降；量化感知训练虽能恢复性能，却需额外训练。将量化推进至三值（2比特）区间可实现更大节省，但长期存在不稳定性问题。基于近期研究显示，无偏置、带RMS归一化的Transformer结合直通估计可达到1.58比特精度，本文证明：仅在每个线性投影前插入RMS归一化，并采用渐进式、逐层量化调度，即可稳定地将全精度检查点微调为三值LLM。该方法在标准语言建模基准上表现匹配或优于更复杂的知识蒸馏流程，且未增加模型复杂度。结果表明，精心设计的归一化机制本身即可弥合三值与全精度模型间的大部分精度差距，使超低比特推理成为现实。
原文摘要 · Abstract (English)
Large language models (LLMs) have transformed natural-language processing, yet their scale makes real-world deployment costly. Post-training quantization reduces memory and computation but often degrades accuracy, while quantization-aware training can recover performance at the cost of extra training. Pushing quantization to the ternary (2-bit) regime yields even larger savings but is notoriously unstable. Building on recent work showing that a bias-free, RMS-normalized Transformer with straight-through estimation can reach 1.58-bit precision, we demonstrate that simply inserting RMS normalization before every linear projection and applying a gradual, layer-wise quantization schedule stably fine-tunes full-precision checkpoints into ternary LLMs. Our approach matches or surpasses more elaborate knowledge-distillation pipelines on standard language-modeling benchmarks without adding model complexity. These results indicate that careful normalization alone can close much of the accuracy gap between ternary and full-precision LLMs, making ultra-low-bit inference practical.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。