用信息熵判断视觉Transformer中哪些块可删,不需数据也能高效压缩模型。
Entropy Reveals Block Importance in Masked Self-Supervised Vision Transformers
- 通过分析预训练权重的熵值估算各模块重要性。
- 剪掉91.7%的模块后仍保持良好迁移性能。
- 适合资源受限场景下的模型压缩与高效微调。
掩码自监督视觉Transformer已成为主流预训练范式,但其庞大的模型规模给资源受限部署和高效迁移学习带来挑战。一个根本问题是:所有Transformer块对下游任务是否同样重要?本文发现,无需任何数据即可准确估计块的重要性——预训练块权重的信息熵与通过迭代移除块并微调获得的敏感度高度相关。基于此提出Gardener,一种无数据、单次、块级剪枝方法,仅通过信息论测量即可识别冗余块。我们在VideoMAE-B上评估Gardener在多种剪枝率和下游视频识别任务上的表现。尽管计算开销极小,其性能始终优于或媲美现有无数据剪枝基线,接近基于敏感度的剪枝结果。令人惊讶的是,即使剪掉高达91.7%的块,模型仍具备竞争力的迁移能力。结果揭示了掩码自监督视觉Transformer存在显著块级冗余,并证明信息论分析为模型压缩与资源高效迁移学习提供了原则性且高效的路径。
原文摘要 · Abstract (English)
Masked self-supervised vision transformers have become a dominant pretraining paradigm, yet their substantial model size poses significant challenges for resource-constrained deployment and efficient transfer learning. A fundamental question remains: are all transformer blocks equally important for downstream performance? In this paper, we show that block importance in masked self-supervised vision transformers can be accurately estimated without access to any data. Our key finding is that the information entropy of pretrained block weights strongly correlates with oracle sensitivity obtained via iterative block removal and finetuning. This observation enables Gardener, a data-free, one-shot, block-level pruning principle that identifies redundant blocks through simple information-theoretic measurements. We evaluate Gardener on VideoMAE-B across multiple pruning ratios and downstream video recognition benchmarks. Despite its negligible computational overhead, Gardener consistently matches or outperforms existing data-free pruning baselines and closely approaches sensitivity-based pruning. Remarkably, even after pruning up to 91.7\% of blocks, the pruned model retains competitive transfer performance. Our results reveal substantial block-level redundancy in masked self-supervised vision transformers and demonstrate that information-theoretic analysis offers a principled and efficient pathway for model compression and resource-efficient transfer learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。