用元网络自动学习剪枝规则,无需训练即可通用剪枝。
Meta Pruning via Graph Metanetworks : A Universal Meta Learning Framework for Network Pruning
- 用图神经网络构建元网络,自动学习复杂剪枝策略。
- 在多种网络上实现顶尖剪枝效果,无需额外训练。
- 适合需要快速、通用剪枝的工程与研究场景。
我们提出一种全新的元学习框架用于网络剪枝。该框架具有高度通用性和可迁移性,理论上可应用于几乎所有类型的网络及剪枝方式。实验表明,其在多种主流代表性剪枝任务(包括CNN和Transformer)中均取得卓越性能。与以往依赖固定手工规则或需每轮剪枝单独训练的方法不同,本框架通过神经网络(元网络)自动学习复杂剪枝规则,且剪枝过程无需特殊训练。具体而言,我们将神经网络与图之间建立双射映射,使用图神经网络作为元网络。训练一个能自动学习剪枝策略的元网络,将难以剪枝的网络转化为更易剪枝的形态。元网络训练完成后,仅需前向传播与标准微调即可完成当前状态最优的剪枝。代码已公开于 https://github.com/Yewei-Liu/MetaPruning。
原文摘要 · Abstract (English)
We propose an entirely new meta-learning framework for network pruning. It is a general framework that can be theoretically applied to almost all types of networks with all kinds of pruning and has great generality and transferability. Experiments have shown that it can achieve outstanding results on many popular and representative pruning tasks (including both CNNs and Transformers). Unlike all prior works that either rely on fixed, hand-crafted criteria to prune in a coarse manner, or employ learning to prune ways that require special training during each pruning and lack generality. Our framework can learn complex pruning rules automatically via a neural network (metanetwork) and has great generality that can prune without any special training. More specifically, we introduce the newly developed idea of metanetwork from meta-learning into pruning. A metanetwork is a network that takes another network as input and produces a modified network as output. In this paper, we first establish a bijective mapping between neural networks and graphs, and then employ a graph neural network as our metanetwork. We train a metanetwork that learns the pruning strategy automatically and can transform a network that is hard to prune into another network that is much easier to prune. Once the metanetwork is trained, our pruning needs nothing more than a feedforward through the metanetwork and some standard finetuning to prune at state-of-the-art. Our code is available at https://github.com/Yewei-Liu/MetaPruning
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。