通过微调内部状态，用极少量数据就能高效恢复剪枝模型的精度。
STARFISH: faST Accuracy Recovery in pruned networks From Internal State Healing
- 利用少量无标签数据对齐剪枝网络与原网络的内部状态。
- 剪掉75%权重后仍能恢复82%原始精度，超领先方法42个百分点。
- 特别适合极端剪枝场景，仅需0.4%训练数据即可生效。
剪枝旨在减少大型神经网络的参数量，虽能显著提升推理速度，但常导致精度大幅下降，通常需通过修复过程恢复部分性能。本文提出一种新修复方法STARFISH，可高效恢复任意剪枝网络的精度。其核心思想是使用极小规模的无标签校准集，优化剪枝网络以对齐原始网络的内部状态表示。在移除50%权重的ViT类网络上，STARFISH相较现有最优方法最高提升22%恢复精度；在极端剪枝下优势更明显：对于ImageNet上的DeiT-B网络，剪掉75%权重后，仅需0.4%的训练图像作为校准集，即可恢复82%的原始密集模型精度，而其他方法最高仅达40%。
原文摘要 · Abstract (English)
Pruning is a process designed to reduce the number of weights in a large neural network. This can substantially speed up inference but might cause a considerable reduction in the model's accuracy, and thus it is usually followed by a healing process that regains some of the lost accuracy. In this paper, we propose a new healing method, STARFISH, that can recover (most of) the accuracy of any pruned network efficiently. The main idea of STARFISH is to optimize the pruned network to align with the original network's internal state representations using a tiny calibration set of unlabeled examples. For the common case of removing 50% of the weights, STARFISH healing improves the recovered accuracy by up to 22% over the state-of-the-art methods on ViT-based networks. Its advantage is even more pronounced under aggressive pruning. For example, after eliminating 75% of the weights in a DeiT-B network for ImageNet, STARFISH uses only 0.4% of the number of training images as a calibration set and recovers 82% of the original dense accuracy, whereas competing recovery techniques reach only 40% of the dense model accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。