无需微调，一次剪枝即可高效压缩Transformer模型。
CORP: Closed-Form One-shot Representation-Preserving Structured Pruning for Transformers
- 通过闭式解直接计算剪枝补偿，不依赖梯度或重训练。
- 在DeiT-Huge上剪掉50%的MLP与注意力结构，仍保持83.27%准确率。
- 适合部署前轻量化模型，尤其适用于资源受限场景。
Transformer模型虽性能优异，但计算和内存开销大。结构化剪枝可降低推理成本，但多数方法依赖微调或多阶段优化，限制了训练后部署。本文提出CORP，一种闭式单次结构化剪枝方法，仅使用无标签校准数据即可移除MLP维度和注意力子结构，无需梯度或微调。CORP将结构化剪枝建模为表示恢复问题，将被移除组件视为保留组件的仿射函数，推导出闭式岭回归解，将补偿融合至模型权重中，最小化校准分布下的层内仿射/逻辑重构目标。ImageNet上的实验表明MLP和注意力表示存在显著冗余。在DeiT上，即使在激进稀疏度下，模型仍保持高精度。在DeiT-Huge上，剪枝50%的MLP与注意力结构后，仍达到83.27%的Top-1准确率。
原文摘要 · Abstract (English)
Transformers achieve strong accuracy but incur high compute and memory cost. Structured pruning reduces inference cost, but most methods rely on retraining or multi-stage optimization, which limits post-training deployment. We propose CORP, a closed-form one-shot structured pruning method that removes MLP dimensions and attention substructures using only unlabeled calibration data without gradients or fine-tuning. CORP formulates structured pruning as a representation recovery problem. It models removed components as affine functions of retained components and derives closed-form ridge regression solutions that fold compensation into model weights. This minimizes a layer-local affine/logit reconstruction objective under the calibration distribution. Experiments on ImageNet with DeiT reveal strong redundancy in both MLP and attention representations. With CORP, models retain high accuracy under aggressive sparsity. On DeiT-Huge, CORP achieves 83.27% Top-1 accuracy after pruning 50\% of both MLP and attention structures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。