用图曲率识别神经网络中冗余连接,提升剪枝效率。
Post-Training Neural Network Pruning using Graph Curvature
- 基于图曲率构建神经网络连接重要性评分体系。
- 在多个数据集上剪枝后模型性能下降小于1%,且剪枝率更高。
- 适合追求高精度剪枝的工程应用与模型压缩研究者。
本文从图论视角重新审视神经网络剪枝问题。为实现高效剪枝,我们旨在识别神经网络中的主要数据流及其对应连接的重要性。不同于传统基于信息论的数据流分析方法,我们采用图曲率概念,特别是Ollivier-Ricci曲率(ORC)。ORC已被成功应用于道路流量、生物网络和社交网络等领域的关键边识别:负曲率边被视为瓶颈,对整体连通性至关重要;正曲率边则较不重要。我们将此思想引入神经网络,首先构建由神经网络结构诱导的图,并基于ORC定义神经曲率(NC);其次,基于一组输入样本的激活模式计算曲率;最后证明NC可用于按连接对整体网络功能的重要性进行排序。我们在多个小型和中型模型上进行剪枝实验,这些模型在MNIST、CIFAR-10和CIFAR-100三个图像数据集上训练。结果表明,该方法相比现有剪枝方法能识别出更多不重要的连接。
原文摘要 · Abstract (English)
This paper provides a fresh view of the neural network (NN) pruning problem through the lens of graph theory. To achieve effective pruning, we aim to identify the main NN data flows and the corresponding NN connections that are most and least important for the performance of the full model. Unlike the standard approach to NN data flow analysis, which is based on information theory, we employ the notion of graph curvature, specifically Ollivier-Ricci curvature (ORC). ORC has been successfully used to identify important graph edges in various domains such as road traffic analysis, biological networks, and social networks. In particular, edges with negative ORC are considered bottlenecks and are therefore critical to the graph's overall connectivity, whereas positive-ORC edges are less essential. We use this intuition for NNs to (1) construct a graph induced by the NN structure and introduce the notion of neural curvature (NC) based on ORC; (2) calculate curvatures based on activation patterns for a set of input examples; and (3) demonstrate that NC can be used to rank edges according to their importance for overall NN functionality. We evaluate our method through pruning experiments on a variety of small and medium size models trained on three image datasets: MNIST, CIFAR-10, and CIFAR-100. The results indicate that our method can identify a larger number of unimportant edges compared to existing pruning methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。