通过结构化稀疏与低精度量化协同压缩模型,保持高精度。
Extreme Model Compression with Structured Sparsity at Low Precision
- 训练时用角度对齐正则化,协调稀疏与量化影响。
- ResNet-18压缩20倍,准确率保留99%以上。
- 适用于图像分类、检测、分割等多种任务。
深度神经网络在诸多应用中被广泛使用,但其庞大的规模和高昂的计算成本使其难以在资源受限设备上运行。常用的两种解决方案是权重量化(降低所有权重的精度)和结构化稀疏(移除不重要的权重,保留重要权重于全精度)。尽管两者单独有效,但因联合使用会显著损害模型准确率,通常被分开研究。本文提出SLOPE(Structured Sparsity at Low Precision),一种统一框架,以系统方式结合结构化稀疏与低比特量化。我们发现,简单合并稀疏与量化会严重损害性能。为此,提出一种训练阶段正则化策略,通过促进全精度权重与其稀疏量化版本之间的角度对齐,而非直接匹配,来最小化差异。在ResNet-18上,SLOPE实现约20倍模型尺寸压缩,同时保留约99%原始准确率。该方法在分类、检测和分割任务中,对ResNet-18、ViT-Small和Mask R-CNN等模型均持续优于当前最优的量化和结构化稀疏方法。
原文摘要 · Abstract (English)
Deep neural networks (DNNs) are used in many applications, but their large size and high computational cost make them hard to run on devices with limited resources. Two widely used techniques to address this challenge are weight quantization, which lowers the precision of all weights, and structured sparsity, which removes unimportant weights while retaining the important ones at full precision. Although both are effective individually, they are typically studied in isolation due to their compounded negative impact on model accuracy when combined. In this work, we introduce SLOPE Structured Sparsity at Low Precision), a unified framework, to effectively combine structured sparsity and low-bit quantization in a principled way. We show that naively combining sparsity and quantization severely harms performance due to the compounded impact of both techniques. To address this, we propose a training-time regularization strategy that minimizes the discrepancy between full-precision weights and their sparse, quantized counterparts by promoting angular alignment rather than direct matching. On ResNet-18, SLOPE achieves $\sim20\times$ model size reduction while retaining $\sim$99% of the original accuracy. It consistently outperforms state-of-the-art quantization and structured sparsity methods across classification, detection, and segmentation tasks on models such as ResNet-18, ViT-Small, and Mask R-CNN.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。