用二次二值表达式压缩矩阵,比传统方法更省内存且精度更高
Binary Quadratic Quantization: Beyond First-Order Quantization for Real-Valued Matrix Compression
- 用二值二次项替代线性组合,提升矩阵表达能力
- 在2比特量化下,图像分类准确率领先主流方法2.2%~59.1%
- 无需专门优化,即可实现高效模型压缩,适合部署场景
本文提出一种新型矩阵量化方法——二值二次量化(Binary Quadratic Quantization, BQQ)。与传统的线性二值基逼近方法(如均匀量化、二值编码量化)不同,BQQ利用二值二次表达式的强大表示能力,在保持极紧凑数据格式的同时实现更优的矩阵压缩。我们通过两个实验验证该方法:矩阵压缩基准测试和基于预训练视觉变换器模型的后训练量化(PTQ)。结果表明,BQQ在多种矩阵数据上均优于传统方法,实现了内存效率与重建误差之间的更优平衡。此外,其在PTQ任务中表现优异,尽管未针对严苛内存约束下的最优精度进行设计,也未依赖特定于PTQ的二值矩阵优化。例如,在图像分类任务中,本方法在基于校准和无数据场景下分别超越当前最优PTQ方法2.2%和59.1%,量化位宽等效为2比特。这些发现揭示了二值二次表达式在高效矩阵近似与神经网络压缩中的惊人潜力。
原文摘要 · Abstract (English)
This paper proposes a novel matrix quantization method, Binary Quadratic Quantization (BQQ). In contrast to conventional first-order quantization approaches, such as uniform quantization and binary coding quantization, that approximate real-valued matrices via linear combinations of binary bases, BQQ leverages the expressive power of binary quadratic expressions while maintaining an extremely compact data format. We validate our approach with two experiments: a matrix compression benchmark and post-training quantization (PTQ) on pretrained Vision Transformer-based models. Experimental results demonstrate that BQQ consistently achieves a superior trade-off between memory efficiency and reconstruction error than conventional methods for compressing diverse matrix data. It also delivers strong PTQ performance, even though we neither target state-of-the-art PTQ accuracy under tight memory constraints nor rely on PTQ-specific binary matrix optimization. For example, our proposed method outperforms the state-of-the-art PTQ method by up to 2.2\% and 59.1% on the ImageNet dataset under the calibration-based and data-free scenarios, respectively, with quantization equivalent to 2 bits. These findings highlight the surprising effectiveness of binary quadratic expressions for efficient matrix approximation and neural network compression.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。