极端量化模型生成质量下降，源于平滑性损失而非精度问题。
Fitting Is Not Enough: Smoothness in Extremely Quantized LLMs
- 提出平滑性代理指标，揭示低比特量化导致生成不连贯。
- 发现量化后预测邻域有效词候选急剧减少，导致解码树稀疏。
- 引入平滑性保持机制，显著提升生成质量，适合模型压缩研究者。
大语言模型性能强但部署成本高，推动极低比特但有损量化的发展。现有量化方法主要关注前向计算的数值精度以消除性能下降。本文揭示，极端量化模型不仅存在数值精度损失，还出现系统性平滑性退化。通过平滑性代理指标发现，随着量化位宽降低，平滑性退化愈发严重。基于序列邻域建模，我们观察到量化模型在预测邻域内的有效词候选快速减少，直接导致解码树稀疏和生成质量下降。为验证此现象，我们在后训练量化与量化感知训练中引入简单平滑性保持原则，证明在保证数值精度基础上，保留平滑性可带来额外性能增益。本文核心目标是强调平滑性保持应成为未来极端量化方法的重要设计考量。代码已开源：https://github.com/xuyuzhuang11/FINE。
原文摘要 · Abstract (English)
Large language models (LLMs) achieve strong performance but incur high deployment costs, motivating extremely low-bit but lossy quantization. Existing quantization algorithms mainly focus on improving the numerical accuracy of forward computation to eliminate performance degradation. In this paper, we show that extremely quantized LLMs suffer from systematic smoothness degradation beyond numerical precision loss. Through a smoothness proxy, we observe that such degradation becomes increasingly severe as the quantization bit-width decreases. Furthermore, based on sequence neighborhood modeling, we find that quantized models exhibit a rapid reduction of effective token candidates within the prediction neighborhood, which directly leads to a sparser decoding tree and degraded generation quality. To validate it, we introduce a simple smoothness-preserving principle in both post-training quantization and quantization-aware training, and demonstrate that preserving smoothness brings additional gains beyond numerical accuracy. The core goal of this paper is to highlight smoothness preservation as an important design consideration for future extreme quantization methods. Code is available at https://github.com/xuyuzhuang11/FINE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。