通过迭代剪枝与结构压缩，让模型变小不降效
Squeeze-Release: Iterative Pruning with Exact Structural Minimization
- 剪枝后执行精确结构重写，保持计算结果不变但体积更小
- 多轮剪枝+释放机制使模型压缩至原大小的1/14.8（ConvNeXt-Tiny）
- 适用于带归一化的残差网络，可推广到Transformer架构
非结构化剪枝会产生稀疏权重张量，但传统实现保留原始张量形状，导致部署模型大小不变。本文提出一种精确的结构重写方法，称为最小化，能将掩码网络转换为更小的密集网络，其前向传播结果在浮点舍入误差范围内与原模型一致。Squeeze-Release循环结合剪枝与最小化，并引入中间释放步骤，将压缩后张量中的零值位置重新激活为微小校准噪声，使原本浪费的容量重新变为可训练参数。多轮迭代利用该容量挖掘单次剪枝无法发现的结构冗余。此外，我们提出CompensatedLayerNorm，一种函数保持型的LayerNorm替代方案，使最小化可扩展至含LayerNorm的残差流中的通道缩减。Squeeze-Release在全连接网络上使可部署模型缩小至原模型的1/39，在现代CNN（ConvNeXt-Tiny）上缩小至1/14.8，且精度相当。此外证明该重写可推广至Transformer架构。
原文摘要 · Abstract (English)
Unstructured pruning produces sparse weight tensors, but the standard implementation keeps tensor shapes unchanged so the deployed model is no smaller than before pruning. We present an exact structural rewrite, which we call minimization, that converts a masked network into a smaller dense network with the same forward function up to floating-point rounding. The Squeeze-Release cycle iterates pruning and minimization with an intermediate release step that re-enables the exact-zero positions inside the compacted tensors as small calibrated noise, turning otherwise wasted capacity back into trainable parameters. Successive cycles use that capacity to find structural redundancy a single pass cannot reach. We additionally introduce CompensatedLayerNorm, a function-preserving replacement for LayerNorm that extends minimization to channel reduction across LayerNorm-equipped residual streams. Squeeze-Release compresses the deployable network to 39x smaller than the unpruned model on a fully-connected model network and 14.8x smaller on modern CNN (ConvNeXt-Tiny), at comparable accuracy. In addition we prove that the rewrite can be extended to transformer architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。