提出新正则化方法，让剪枝后的网络保持输入输出连通性。
CoNNect: Connectivity-Based Regularization for Structural Pruning
- 基于连通性设计可微正则化项，引导剪枝过程
- 在多个数据集上提升剪枝后模型精度，最高增益1.8%
- 适合需要高连通性的结构化剪枝场景
剪枝旨在提升神经网络的稀疏性，通常可表述为在L0范数约束下最小化损失函数。本文提出CoNNect，一种新型可微正则化方法，确保剪枝后网络中输入层与输出层之间的连通性。理论证明CoNNect可近似实现L0正则化，在不引发层坍缩等缺陷的前提下，保障网络结构的最大连通性。该方法可无缝集成至现有结构化剪枝策略中。数值实验表明，CoNNect能有效改进经典剪枝方法，并显著提升当前最优的一次性剪枝器（如DepGraph和LLM-pruner）的性能。
原文摘要 · Abstract (English)
Pruning encompasses a range of techniques aimed at increasing the sparsity of neural networks (NNs). These techniques can generally be framed as minimizing a loss function subject to an $L_0$ norm constraint. This paper introduces CoNNect, a novel differentiable regularizer for sparse NN training that ensures connectivity between input and output layers. We prove that CoNNect approximates $L_0$ regularization, guaranteeing maximally connected network structures while avoiding issues like layer collapse. Moreover, CoNNect is easily integrated with established structural pruning strategies. Numerical experiments demonstrate that CoNNect can improve classical pruning strategies and enhance state-of-the-art one-shot pruners, such as DepGraph and LLM-pruner.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。