提出首个兼具信息最优与计算高效特性的量化方法，显著提升低比特模型性能。
Boosting Entropy with Bell Box Quantization
- 在输入域实现信息最优量化，输出映射到计算高效的数值域。
- 1比特模型困惑度降低18点，4比特模型降低2点，性能超越现有最佳方法。
- 适合追求极致压缩与能效的边缘设备模型部署场景。
量化感知预训练（QAPT）能有效降低深度神经网络的计算与内存开销，提升边缘设备的能效。现有QAPT方法虽采用计算高效的存储格式（如整数），但并非信息理论最优（ITO）。而现有的ITO数据类型（如分位数/正态浮点量化）又缺乏计算效率。本文提出BBQ，首个兼具信息理论最优与计算高效的量化方法。其核心洞察是：学习具有领域无关性，量化器输出无需与输入同域。BBQ在输入域执行信息最优量化，并将结果映射至计算高效的数值域，使ITO数据类型适配高效计算格式。在不牺牲计算效率的前提下，BBQ在4位模型上困惑度降低最多2点，3位模型最多4点，2位模型最多5点，1位模型最多18点。代码已开源：https://github.com/1733116199/bbq。
原文摘要 · Abstract (English)
Quantization-Aware Pre-Training (QAPT) is an effective technique to reduce the compute and memory overhead of Deep Neural Networks while improving their energy efficiency on edge devices. Existing QAPT methods produce models stored in compute-efficient data types (e.g. integers) that are not information theoretically optimal (ITO). On the other hand, existing ITO data types (e.g. Quantile/NormalFloat Quantization) are not compute-efficient. We propose BBQ, the first ITO quantization method that is also compute-efficient. BBQ builds on our key insight that since learning is domain-agnostic, the output of a quantizer does not need to reside in the same domain as its input. BBQ performs ITO quantization in its input domain, and returns its output in a compute-efficient domain where ITO data types are mapped to compute-efficient data types. Without sacrificing compute efficiency, BBQ outperforms prior SOTA QAPT methods by a perplexity reduction of up to 2 points for 4-bit models, up to 4 points for 3-bit models, up to 5 points for 2-bit models, and up to 18 points for 1-bit models. Code is available at https://github.com/1733116199/bbq.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。