用小波变换提升1比特量化精度，让大模型压缩后仍保持高保真。
HBLLM: Wavelet-Enhanced High-Fidelity 1-Bit Quantization for LLMs
- 通过小波分解增强权重表达能力，分频段优化参数分组。
- 在LLaMA2-13B上实现6.71的困惑度，平均存储仅1.08位。
- 适合追求极致压缩比且需保持性能的部署场景。
我们提出HBLLM，一种基于小波增强的1比特后训练量化方法，用于大型语言模型（LLMs）。利用哈尔小波变换进行频率分解，显著提升量化保真度，同时保持极低开销。该方法包含两项创新的结构感知分组策略：(1) 频率感知的多参数行内分组，(2) 基于ℓ₂范数显著性驱动的列选择。对非显著权重，在每个频带内采用共享均值以优化存储效率。在OPT和LLaMA模型上的实验表明，HBLLM在1比特量化中达到最先进性能，在LLaMA2-13B上实现6.71的困惑度，平均权重存储仅为1.08比特。代码已公开：https://github.com/Yeyke/HBLLM。
原文摘要 · Abstract (English)
We introduce HBLLM, a wavelet-enhanced high-fidelity $1$-bit post-training quantization method for Large Language Models (LLMs). By leveraging Haar wavelet transforms to enhance expressive capacity through frequency decomposition, HBLLM significantly improves quantization fidelity while maintaining minimal overhead. This approach features two innovative structure-aware grouping strategies: (1) frequency-aware multi-parameter intra-row grouping and (2) $\ell_2$-norm-based saliency-driven column selection. For non-salient weights, a shared mean is employed across quantization groups within each frequency band to optimize storage efficiency. Experiments conducted on the OPT and LLaMA models demonstrate that HBLLM achieves state-of-the-art performance in $1$-bit quantization, attaining a perplexity of $6.71$ on LLaMA$2$-$13$B with an average weight storage of only $1.08$ bits. Code available at: https://github.com/Yeyke/HBLLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。