提出误差理论框架,精准分配模型各层压缩比例,显著减少参数量且不降性能。
A General Error-Theoretical Analysis Framework for Constructing Compression Strategies
- 基于几何结构分析量化误差,构建分层压缩的理论依据。
- 在ResNet-34上实现近11倍参数压缩,性能仍优于原模型。
- 适合追求高效模型部署的研究者与工程师参考。
深度模型参数规模和计算复杂度的指数增长带来了高效部署的巨大挑战。现有压缩方法的核心问题在于:模型不同层对压缩的容忍度差异显著,例如首层可承受更高压缩率而性能损失小,末层则相反。因此,关键挑战是如何在各层间合理分配压缩程度,在最小化性能损失的同时最大化参数压缩。为此,本文提出压缩误差理论(CET)框架,用于确定每层最优压缩水平。以量化为例,CET利用微分扩张与代数几何,将量化误差重构为椭球与双曲抛物面,并通过其几何结构定义误差子空间。通过正交分解几何空间,将误差子空间优化转化为互补问题。理论分析表明,沿主轴构建量化子空间可最小化性能退化。实验验证显示,该理论能大幅保持性能,尤其在ResNet-34上实现近11×参数压缩,且性能超越原模型。
原文摘要 · Abstract (English)
The exponential growth in parameter size and computational complexity of deep models poses significant challenges for efficient deployment. The core problem of existing compression methods is that different layers of the model have significant differences in their tolerance to compression levels. For instance, the first layer of a model can typically sustain a higher compression level compared to the last layer without compromising performance. Thus, the key challenge lies in how to allocate compression levels across layers in a way that minimizes performance loss while maximizing parameter reduction. To address this challenge, we propose a Compression Error Theory (CET) framework, designed to determine the optimal compression level for each layer. Taking quantization as an example, CET leverages differential expansion and algebraic geometry to reconstruct the quadratic form of quantization error as ellipsoids and hyperbolic paraboloids, and utilizes their geometric structures to define an error subspace. To identify the error subspace with minimal performance loss, by performing orthogonal decomposition of the geometric space, CET transforms the optimization process of the error subspace into a complementary problem. The final theoretical analysis shows that constructing the quantization subspace along the major axis results in minimal performance degradation. Through experimental verification of the theory, CET can greatly retain performance while compressing. Specifically, on the ResNet-34 model, CET achieves nearly 11$\times$ parameter compression while even surpassing performance comparable to the original model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。