发现模型收敛后量化性能突然崩溃，源于INT4精度的固有缺陷。
When Flat Minima Fail: Characterizing INT4 Quantization Collapse After FP32 Convergence
- 用无校准的分组INT4探测器分析154个检查点，揭示三阶段演化过程。
- FP32收敛后，INT4误差从11%飙升至517%，而困惑度几乎不变。
- 证明是INT4量化网格粗糙导致，非权重异常或学习率衰减引起。
训练后量化（PTQ）假设充分收敛的模型即为可量化的模型。本文表明该假设在结构化、可测量且此前未被识别的方式下失效。通过在全部154个公开的Pythia-160m训练检查点上应用无校准的分组INT4探测器，我们识别出三阶段分歧结构：快速学习期，FP32困惑度与量化鲁棒性同步提升；中期约7万步的元稳定平台期，FP32困惑度停滞但INT4差距保持有限；以及爆炸式分歧期，此时INT4差距从11%激增至517%，而FP32困惑度变化极小。关键发现是，分歧并非始于学习率衰减，而是当FP32困惑度完成精细收敛时——这表明收敛后的权重更新才是近因，而非衰减幅度。我们进一步验证了INT8在整个过程中完全免疫，将机制锁定于16级INT4网格的粗糙性，并通过直接峰度测量排除了权重异常累积的可能。最后，从分歧前检查点发起受控分叉实验，比较三种学习率调度策略（余弦延续、SGDR热重启、新提出的振荡锁定法）在九次独立运行中的表现。结果表明，SGDR均加速分歧（0/9胜于余弦），而OLI的稳定冷却阶段平均降低INT4差距2.2个百分点（t = -5.46, p < 0.0001），说明调度幅度校准而非振荡本身决定扰动是否有益。所有代码、探测实现及154个检查点审计结果均已公开。
原文摘要 · Abstract (English)
Post-training quantization (PTQ) assumes that a well-converged model is a quantization-ready model. We show this assumption fails in a structured, measurable, and previously uncharacterized way. Using a calibration-free per-group INT4 probe applied to all 154 publicly available Pythia-160m training checkpoints, we identify a three-phase divergence structure: a rapid-learning phase where both FP32 perplexity and quantization robustness improve together, a meta-stable plateau lasting roughly 70,000 steps where FP32 perplexity stagnates but INT4 gap remains bounded, and an explosive divergence phase where the INT4 gap compounds from 11% to 517% while FP32 perplexity barely moves. Critically, this divergence begins not when the learning rate starts decaying, but precisely when FP32 perplexity converges a finer-grained onset predictor that implies post-convergence weight updates, rather than decay magnitude alone, are the proximate cause. We further show that INT8 quantization is entirely immune throughout all three phases, constraining the mechanism to the coarseness of the 16-level INT4 grid specifically, and rule out weight outlier accumulation as the mechanism via direct kurtosis measurement. Finally, we conduct a controlled fork experiment from the pre-divergence checkpoint comparing three learning rate schedules (cosine continuation, SGDR warm restarts, and our proposed Oscillatory Lock-In) across nine independent runs. SGDR uniformly accelerates divergence (0/9 pairwise wins against cosine), while OLI's settled cool phases reduce the INT4 gap by 2.2 percentage points on average (t = -5.46, p < 0.0001), demonstrating that schedule amplitude calibration, not oscillation alone, determines whether perturbation helps or hurts. Our code, probe implementation, and all 154-checkpoint audit results are released publicly.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。