提出新剪枝方法OBA，用精确的海森矩阵计算提升模型效率
Optimal Brain Apoptosis
- 基于海森矩阵直接计算参数重要性，避免近似误差
- 在VGG19、ResNet、ViT等模型上实现高精度剪枝，性能损失<1%
- 适用于复杂网络如CNN和Transformer，适合追求高效部署的研究者
卷积神经网络（CNN）和Transformer的参数量持续增长，带来计算效率与资源消耗的挑战。剪枝作为有效缓解手段，通过移除冗余神经元、通道或连接来提升效率且不显著影响性能。本文在经典最优脑损伤（OBD）基础上，提出一种新剪枝方法——最优脑凋亡（OBA），通过直接计算每个参数的海森向量积来估计参数重要性，无需依赖近似。通过分解各层海森矩阵并识别跨层子矩阵非零条件，提出高效计算参数二阶泰勒展开的技术。实验验证了该方法在VGG19、ResNet32、ResNet50和ViT-B/16上的有效性，涵盖CIFAR10、CIFAR100和ImageNet数据集，结果表明可实现高精度剪枝且性能下降小于1%。代码已开源。
原文摘要 · Abstract (English)
The increasing complexity and parameter count of Convolutional Neural Networks (CNNs) and Transformers pose challenges in terms of computational efficiency and resource demands. Pruning has been identified as an effective strategy to address these challenges by removing redundant elements such as neurons, channels, or connections, thereby enhancing computational efficiency without heavily compromising performance. This paper builds on the foundational work of Optimal Brain Damage (OBD) by advancing the methodology of parameter importance estimation using the Hessian matrix. Unlike previous approaches that rely on approximations, we introduce Optimal Brain Apoptosis (OBA), a novel pruning method that calculates the Hessian-vector product value directly for each parameter. By decomposing the Hessian matrix across network layers and identifying conditions under which inter-layer Hessian submatrices are non-zero, we propose a highly efficient technique for computing the second-order Taylor expansion of parameters. This approach allows for a more precise pruning process, particularly in the context of CNNs and Transformers, as validated in our experiments including VGG19, ResNet32, ResNet50, and ViT-B/16 on CIFAR10, CIFAR100 and Imagenet datasets. Our code is available at https://github.com/NEU-REAL/OBA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。