2bit量化崩溃主因是初始化不佳，新方法显著提升极端压缩效果
Initialisation Determines the Basin: Efficient Codebook Optimisation for Extreme LLM Quantization
- 提出基于海森加权马氏距离的输出感知初始化方法
- 在2比特下使困惑度下降数量级，优于传统搜索与微调
- 适用于边缘部署的极低比特大模型压缩，尤其适合资源受限场景
加性量化实现零开销查表解量化，适合边缘部署。但在2比特精度下，即使经过大量搜索和微调仍常出现灾难性失败。我们发现主要瓶颈在于码本初始化。贪心序列初始化常将模型置于难以优化的区域，后续束搜索与PV微调难以克服。通过表示率ρ = N/KM分析该现象，并提出OA-EM方法，利用海森加权马氏距离进行输出感知初始化。在三种架构（Llama 3.2 3B、Llama 3.1 8B、Qwen 2.5 3B）上，不同压缩率和搜索预算下，OA-EM经PV微调后始终表现更优，全面占据质量-计算权衡前沿。瓶颈严重程度随ρ增大：3bpp时中等，2bpp时极端，不良初始化可使困惑度下降一个数量级。结果凸显压缩模型空间中优化几何的重要性，初始化可主导后续搜索与微调。
原文摘要 · Abstract (English)
Additive quantization enables extreme LLM compression with O(1) lookup-table dequantization, making it attractive for edge deployment. Yet at 2-bit precision, it often fails catastrophically, even with extensive search and finetuning. We show that the dominant bottleneck is codebook initialisation. Greedy sequential initialisation frequently places the model in poor optimisation regions that subsequent beam search and PV-tuning struggle to overcome. We analyse this behaviour through the representational ratio \r{ho} = N/KM, which characterises the relationship between weight groups and codebook capacity, and propose OA-EM, an output-aware EM initialisation method using Hessian-weighted Mahalanobis distance. Across compression rates, search budgets, and three architectures (Llama 3.2 3B, Llama 3.1 8B, Qwen 2.5 3B), OA-EM consistently produces better solutions after PV-tuning and dominates the quality-compute frontier. The severity of the bottleneck scales with \r{ho}: moderate at 3 bpp but extreme at 2 bpp, where poor initialisation can degrade perplexity by orders of magnitude. More broadly, our results highlight the importance of optimisation geometry in compressed model spaces, where initialisation can dominate subsequent search and fine-tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。