提出通用稳定无损压缩框架,让模型变小不丢性能。
Compression for Better: A General and Stable Lossless Compression Framework
- 用总微分界定压缩范围,明确无损压缩的误差边界。
- 在多种模型上实现无损量化与低秩压缩,性能零损失。
- 适合追求高效部署且不接受精度下降的研究者。
本文聚焦于如何实现稳定且无损的模型压缩,旨在降低模型复杂度并提升效率,同时避免因压缩引入误差导致性能下降。当前缺乏系统性方法来确定压缩误差边界及其对模型性能的具体影响。为此,我们提出通用的无损压缩理论框架(LLC),通过总微分进一步划分压缩邻域和高阶分析边界,精确界定模型可无损压缩的误差范围。为验证其有效性,我们在多个神经网络架构和数据集上应用了量化与分解等压缩技术。对于量化,将经典量化搜索问题重构为损失邻域内的分组背包问题,实现无损量化并提升计算效率;对于分解,针对低秩约束下的近似问题,自动确定每层秩,生成无损低秩模型。实验表明,无需复杂技巧,LLC即可有效实现无损压缩。代码将公开。
原文摘要 · Abstract (English)
This work focus on how to stabilize and lossless model compression, aiming to reduce model complexity and enhance efficiency without sacrificing performance due to compression errors. A key challenge is effectively leveraging compression errors and defining the boundaries for lossless compression to minimize model loss. i.e., compression for better. Currently, there is no systematic approach to determining this error boundary or understanding its specific impact on model performance. We propose a general \textbf{L}oss\textbf{L}ess \textbf{C}ompression theoretical framework (\textbf{LLC}), which further delineates the compression neighborhood and higher-order analysis boundaries through the total differential, thereby specifying the error range within which a model can be compressed without loss. To verify the effectiveness of LLC, we apply various compression techniques, including quantization and decomposition. Specifically, for quantization, we reformulate the classic quantization search problem as a grouped knapsack problem within the lossless neighborhood, achieving lossless quantization while improving computational efficiency. For decomposition, LLC addresses the approximation problem under low-rank constraints, automatically determining the rank for each layer and producing lossless low-rank models. We conduct extensive experiments on multiple neural network architectures on different datasets. The results show that without fancy tricks, LLC can effectively achieve lossless model compression. Our code will be made publicly.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。