将扩散模型压缩为1比特并单步推理,实现超分辨率在边缘设备的高效部署。
BiMaCoSR: Binary One-Step Diffusion Model Leveraging Flexible Matrix Compression for Real Super-Resolution
- 结合二值化与单步推理,实现极致压缩与加速。
- 提出稀疏与低秩矩阵分支,防止二值化导致性能崩溃。
- 在边缘设备上实现23.8倍压缩和27.4倍加速,适合资源受限场景。
基于扩散模型(DM)的超分辨率(SR)方法虽表现优异,但因内存与计算开销大,难以部署于资源受限的边缘设备。现有研究通过二值化降低存储与计算压力,或通过多步到单步蒸馏提升推理速度,但无法满足极端轻量化需求。为此,本文提出BiMaCoSR,融合二值化与单步蒸馏,实现极致压缩与加速。为避免二值化带来的性能坍塌,设计稀疏矩阵分支(SMB)与低秩矩阵分支(LRMB),二者均传递全精度(FP)信息:SMB捕捉极端值,输出高秩表征;LRMB受LoRA启发,初始化为前r个SVD分量,输出低秩表示。两分支计算与存储开销可忽略。实验表明,BiMaCoSR优于当前主流二值化方法,且与全精度单步模型性能相当。相比全精度模型,实现23.8倍压缩比与27.4倍加速比。代码与模型已开源。
原文摘要 · Abstract (English)
While super-resolution (SR) methods based on diffusion models (DM) have demonstrated inspiring performance, their deployment is impeded due to the heavy request of memory and computation. Recent researchers apply two kinds of methods to compress or fasten the DM. One is to compress the DM into 1-bit, aka binarization, alleviating the storage and computation pressure. The other distills the multi-step DM into only one step, significantly speeding up inference process. Nonetheless, it remains impossible to deploy DM to resource-limited edge devices. To address this problem, we propose BiMaCoSR, which combines binarization and one-step distillation to obtain extreme compression and acceleration. To prevent the catastrophic collapse of the model caused by binarization, we proposed sparse matrix branch (SMB) and low rank matrix branch (LRMB). Both auxiliary branches pass the full-precision (FP) information but in different ways. SMB absorbs the extreme values and its output is high rank, carrying abundant FP information. Whereas, the design of LRMB is inspired by LoRA and is initialized with the top r SVD components, outputting low rank representation. The computation and storage overhead of our proposed branches can be safely ignored. Comprehensive comparison experiments are conducted to exhibit BiMaCoSR outperforms current state-of-the-art binarization methods and gains competitive performance compared with FP one-step model. BiMaCoSR achieves a 23.8x compression ratio and a 27.4x speedup ratio compared to FP counterpart. Our code and model are available at https://github.com/Kai-Liu001/BiMaCoSR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。