不依赖真实数据，用生成图像恢复剪枝后模型精度
Post-Pruning Accuracy Recovery via Data-Free Knowledge Distillation
- 用反向归一化统计生成隐私保护的虚拟图像
- 在CIFAR-10上实现剪枝后精度显著恢复
- 适合医疗金融等数据受限场景
模型剪枝能有效降低深度神经网络的计算复杂度和内存占用，但全局非结构化剪枝常导致精度大幅下降，通常需在原始训练数据上微调以恢复性能。在医疗、金融等隐私敏感领域，部署后往往因法规（如GDPR、HIPAA）无法获取原始训练数据。本文提出一种无数据知识蒸馏框架，利用DeepInversion技术从预训练教师模型中反推批次归一化统计，生成隐私保护的‘梦境’图像作为转移数据集，将知识从原教师模型迁移到剪枝后的学生网络。在多种架构（ResNet、MobileNet、VGG）上的CIFAR-10实验表明，该方法无需访问任何真实数据点即可显著恢复剪枝造成的精度损失。
原文摘要 · Abstract (English)
Model pruning is a widely adopted technique to reduce the computational complexity and memory footprint of Deep Neural Networks (DNNs). However, global unstructured pruning often leads to significant degradation in accuracy, typically necessitating fine-tuning on the original training dataset to recover performance. In privacy-sensitive domains such as healthcare or finance, access to the original training data is often restricted post-deployment due to regulations (e.g., GDPR, HIPAA). This paper proposes a Data-Free Knowledge Distillation framework to bridge the gap between model compression and data privacy. We utilize DeepInversion to synthesize privacy-preserving ``dream'' images from the pre-trained teacher model by inverting Batch Normalization (BN) statistics. These synthetic images serve as a transfer set to distill knowledge from the original teacher to the pruned student network. Experimental results on CIFAR-10 across various architectures (ResNet, MobileNet, VGG) demonstrate that our method significantly recovers accuracy lost during pruning without accessing a single real data point.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。