预训练模型剪枝无需任务数据，仍可保持跨任务泛化能力。
A Second-Order Perspective on Pruning at Initialization and Knowledge Transfer
- 在初始化时剪枝，不依赖下游任务数据
- 剪枝后模型零样本迁移性能仍稳定
- 微调可恢复未见任务的性能，适合通用部署
预训练视觉模型的广泛可用性推动了深度学习应用的发展，但其计算和存储开销限制了实际部署。初始化剪枝作为一种在训练前压缩模型的有前景方法，支持高效的任务特定适配。传统观点认为有效剪枝需依赖任务特定数据，这在下游任务未知时带来挑战。本文研究数据对预训练视觉模型剪枝的影响，发现仅在一个任务上剪枝，模型在未见任务上仍保持零样本性能。此外，微调这些剪枝模型不仅能提升原任务表现，还能恢复未见任务的性能。我们归因于大规模数据预训练所形成的有利损失曲面。
原文摘要 · Abstract (English)
The widespread availability of pre-trained vision models has enabled numerous deep learning applications through their transferable representations. However, their computational and storage costs often limit practical deployment. Pruning-at-Initialization has emerged as a promising approach to compress models before training, enabling efficient task-specific adaptation. While conventional wisdom suggests that effective pruning requires task-specific data, this creates a challenge when downstream tasks are unknown in advance. In this paper, we investigate how data influences the pruning of pre-trained vision models. Surprisingly, pruning on one task retains the model's zero-shot performance also on unseen tasks. Furthermore, fine-tuning these pruned models not only improves performance on original seen tasks but can recover held-out tasks' performance. We attribute this phenomenon to the favorable loss landscapes induced by extensive pre-training on large-scale datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。