基于重要性分数自动确定可剪枝数量，无需手动设阈值。
Effective Model Pruning: Measure The Redundancy of Model Components
- 用有效样本量原理从分数分布中自适应生成剪枝阈值。
- 剪掉低分组件后，性能损失有理论上限保证。
- 适用于各类模型和不同重要性指标，通用性强。
本文探讨模型剪枝中的基础问题：给定模型组件的重要性分数向量 $s$，有多少组件可被移除而不影响性能？提出有效模型剪枝（EMP），利用粒子滤波中的有效样本量（即逆辛普森指数）直接从分数分布推导出所需稀疏度。不预设剪枝标准，而是将 $s$ 映射为有效样本数 $N_{eff}=N_{eff}(s)$，保留前 $N_{eff}$ 个高分组件，剪掉其余 $N-N_{eff}$ 个低分组件。推导出保留分数总和 $s_{eff}$ 与 $N_{eff}$ 的紧下界。该方法确保剪枝后模型损失变化存在理论上限。实验在MLP、CNN、Transformer、LLM及KAN等多种架构上验证了其有效性，并能兼容权重大小、注意力分数、KAN重要性、甚至图像像素等特征级信号的剪枝需求。
原文摘要 · Abstract (English)
This article initiates the study of a basic question about model pruning. Given a vector $s$ of importance scores assigned to model components, how many of the scored components could be discarded without sacrificing performance? We propose Effective Model Pruning (EMP), which derives the desired sparsity directly from the score distribution using the notion of effective sample size from particle filtering, also known as the inverse Simpson index. Rather than prescribe a pruning criterion, EMP supplies a universal adaptive threshold derived from the distribution of the score $s$ over the model components: EMP maps $s$ to a number $N_{eff}=N_{eff}(s)$, called the effective sample size. The $N-N_{eff}$ lowest scoring components are discarded. A tight lower bound on the effective mass $s_{eff}$ (the sum of retained normalized scores) in terms of $N_{eff}$ is derived. This process yields models with a provable upper bound on the loss change relative to the original dense model. Numerical experiments are performed demonstrating this phenomenon across a variety of network architectures including MLPs, CNNs, Transformers, LLMs, and KAN. It is also shown that EMP addresses a rich set of pruning criteria such as weight magnitude, attention score, KAN importance score, and even feature-level signals such as image pixels.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。