用数据压缩法在初始化时找到稳定稀疏子网络，提升剪枝效率。
Finding Stable Subnetworks at Initialization with Dataset Distillation
- 用压缩数据做迭代剪枝，生成可训练的稀疏子网络。
- 仅用1/150训练样本即达到传统方法性能，适用于ResNet-18/CIFAR-10。
- 首次证明不稳定初始化中也能存在稳定子网络，适合高效剪枝研究者。
近期工作表明，数据集蒸馏可用于加速深度学习模型训练。然而其对训练动态、尤其是神经网络剪枝的影响仍不明确。本文在迭代幅度剪枝的内循环中引入蒸馏数据，于初始化阶段生成可训练的稀疏子网络——即彩票票券（lottery tickets）。尽管仅使用150倍少的训练样本，该方法在ResNet-18与CIFAR-10上性能与传统彩票票券重绕方法相当。已有研究指出，彩票票券存在于对SGD噪声稳定的稠密初始化中（即不同数据顺序训练收敛至相同极小值）。我们拓展此发现，证明即使在不稳定稠密初始化中，稳定子网络依然存在。在线性模式连接研究中，我们发现使用蒸馏数据剪枝会移除导致损失曲面尖锐的参数。最后，通过先生成初始化时的稳定稀疏掩码，我们可在远高于传统迭代幅度剪枝的稀疏度下找到彩票票券。
原文摘要 · Abstract (English)
Recent works have shown that Dataset Distillation, the process for summarizing the training data, can be leveraged to accelerate the training of deep learning models. However, its impact on training dynamics, particularly in neural network pruning, remains largely unexplored. In our work, we use distilled data in the inner loop of iterative magnitude pruning to produce sparse, trainable subnetworks at initialization -- more commonly known as lottery tickets. While using 150x less training points, our algorithm matches the performance of traditional lottery ticket rewinding on ResNet-18 & CIFAR-10. Previous work highlights that lottery tickets can be found when the dense initialization is stable to SGD noise (i.e. training across different ordering of the data converges to the same minima). We extend this discovery, demonstrating that stable subnetworks can exist even within an unstable dense initialization. In our linear mode connectivity studies, we find that pruning with distilled data discards parameters that contribute to the sharpness of the loss landscape. Lastly, we show that by first generating a stable sparsity mask at initialization, we can find lottery tickets at significantly higher sparsities than traditional iterative magnitude pruning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。