arXiv:2511.16653cs.CVcs.AI2025-11中稿 · 2025 IEEE Internat…

用教师模型指导一次性剪枝,高效压缩模型且保持精度。

Teacher-Guided One-Shot Pruning via Context-Aware Knowledge Distillation

  • 结合知识蒸馏梯度,动态评估参数重要性
  • 在高稀疏率下仍保持低精度损失,优于现有方法
  • 适合资源受限场景的快速模型压缩

非结构化剪枝虽能有效压缩深度神经网络,但通常需多次训练-剪枝-再训练循环,计算开销大。本文提出一种新型教师引导剪枝框架,将知识蒸馏(KD)与重要性评分紧密融合。不同于以往将KD作为剪枝后恢复步骤的做法,本方法利用教师模型提供的梯度信号,在计算参数重要性时识别对任务性能和知识迁移至关重要的权重,实现一次性的全局剪枝。剪枝后采用面向稀疏性的再训练策略,无需重新激活被剪掉的连接即可恢复精度。在CIFAR-10、CIFAR-100和TinyImageNet等多个图像分类基准上实验表明,该方法在高稀疏率下均能实现极小精度损失,显著优于EPG、EPSD等先进基线,同时比COLT等迭代剪枝方案更高效。所提框架为资源受限环境下的模型部署提供了高效且性能稳定的压缩解决方案。

原文摘要 · Abstract (English)

Unstructured pruning remains a powerful strategy for compressing deep neural networks, yet it often demands iterative train-prune-retrain cycles, resulting in significant computational overhead. To address this challenge, we introduce a novel teacher-guided pruning framework that tightly integrates Knowledge Distillation (KD) with importance score estimation. Unlike prior approaches that apply KD as a post-pruning recovery step, our method leverages gradient signals informed by the teacher during importance score calculation to identify and retain parameters most critical for both task performance and knowledge transfer. Our method facilitates a one-shot global pruning strategy that efficiently eliminates redundant weights while preserving essential representations. After pruning, we employ sparsity-aware retraining with and without KD to recover accuracy without reactivating pruned connections. Comprehensive experiments across multiple image classification benchmarks, including CIFAR-10, CIFAR-100, and TinyImageNet, demonstrate that our method consistently achieves high sparsity levels with minimal performance degradation. Notably, our approach outperforms state-of-the-art baselines such as EPG and EPSD at high sparsity levels, while offering a more computationally efficient alternative to iterative pruning schemes like COLT. The proposed framework offers a computation-efficient, performance-preserving solution well suited for deployment in resource-constrained environments.

模型压缩知识蒸馏剪枝

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。