提出新剪枝框架，兼顾参数间关联，提升模型压缩效率
Optimal Brain Connection: Towards Efficient Structural Pruning
- 用雅可比准则衡量结构化参数重要性，捕捉层间依赖关系
- 在微调中通过自编码器保留被剪枝连接的贡献，减少性能下降
- 适合需要高精度压缩模型的研究者和部署场景
结构化剪枝被广泛用于压缩神经网络，但现有方法常忽略参数间的相互作用。本文提出名为最优脑连接（Optimal Brain Connection）的剪枝框架。首先，引入雅可比准则（Jacobian Criterion），一种一阶指标，用于评估结构化参数的重要性。与以往孤立评估参数的方法不同，该准则显式建模组件内部交互及层间依赖关系。其次，提出等效剪枝机制（Equivalent Pruning），利用自编码器在微调过程中保留所有原始连接（包括被剪枝的）的贡献，有效缓解微调后的性能退化。实验表明，雅可比准则在保持模型性能方面优于多个主流指标，等效剪枝机制显著降低剪枝后性能损失。代码已开源。
原文摘要 · Abstract (English)
Structural pruning has been widely studied for its effectiveness in compressing neural networks. However, existing methods often neglect the interconnections among parameters. To address this limitation, this paper proposes a structural pruning framework termed Optimal Brain Connection. First, we introduce the Jacobian Criterion, a first-order metric for evaluating the saliency of structural parameters. Unlike existing first-order methods that assess parameters in isolation, our criterion explicitly captures both intra-component interactions and inter-layer dependencies. Second, we propose the Equivalent Pruning mechanism, which utilizes autoencoders to retain the contributions of all original connection--including pruned ones--during fine-tuning. Experimental results demonstrate that the Jacobian Criterion outperforms several popular metrics in preserving model performance, while the Equivalent Pruning mechanism effectively mitigates performance degradation after fine-tuning. Code: https://github.com/ShaowuChen/Optimal_Brain_Connection
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。