提出新压缩准则Expressiveness，无需训练即可评估神经元表达能力。
Neural expressiveness for beyond importance model compression
- 基于激活重叠度衡量神经元信息重分配能力，不依赖训练状态。
- 独立使用可提升压缩效率，与重要性方法结合可增效10倍。
- 在YOLOv8上减少55.4%参数，降低46.1%计算量，精度反升3%。
神经网络剪枝已成为推动内存与能效优化、高吞吐量训练与推理的核心技术。本文提出一种新型模型压缩准则——'表达力（Expressiveness）'，不同于依赖权重重要性的传统剪枝方法，该准则基于激活重叠度，衡量神经元或神经元组有效重分配信息资源的能力。这一特性与网络初始化状态强相关，实现对学习状态的解耦，使压缩策略摆脱'何时剪枝'的依赖，为无数据依赖（Data-Agnostic）剪枝提供基础。我们证明表达力可通过任意数据或有限样本近似，支持通用剪枝策略探索。同时，表达力与重要性剪枝可融合形成混合策略，相比权重剪枝方法实现最高10倍的参数压缩比提升，平均性能损失仅1%。独立使用表达力剪枝亦优于当前主流方法。在YOLOv8上，移除55.4%参数，实现46.1%的计算量（MACs）减少，且在COCO数据集上的平均绝对精度（mAP₅₀₋₉₅）提升3%。
原文摘要 · Abstract (English)
Neural Network Pruning has been established as driving force in the exploration of memory and energy efficient solutions with high throughput both during training and at test time. In this paper, we introduce a novel criterion for model compression, named "Expressiveness". Unlike existing pruning methods that rely on the inherent "Importance" of neurons' and filters' weights, ``Expressiveness" emphasizes a neuron's or group of neurons ability to redistribute informational resources effectively, based on the overlap of activations. This characteristic is strongly correlated to a network's initialization state, establishing criterion autonomy from the learning state stateless and thus setting a new fundamental basis for the expansion of compression strategies in regards to the "When to Prune" question. We show that expressiveness is effectively approximated with arbitrary data or limited dataset's representative samples, making ground for the exploration of Data-Agnostic strategies. Our work also facilitates a "hybrid" formulation of expressiveness and importance-based pruning strategies, illustrating their complementary benefits and delivering up to 10x extra gains w.r.t. weight-based approaches in parameter compression ratios, with an average of 1% in performance degradation. We also show that employing expressiveness (independently) for pruning leads to an improvement over top-performing and foundational methods in terms of compression efficiency. Finally, on YOLOv8, we achieve a 46.1% MACs reduction by removing 55.4\% of the parameters, with an increase of 3% in the mean Absolute Precision ($mAP_{50-95}$) for object detection on COCO dataset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。