用随机矩阵理论剪枝，小调优就能保持高精度。
Pruning Deep Neural Networks via the Marchenko--Pastur Distribution
- 基于马尔琴科-帕斯图分布设计剪枝准则，无需长周期微调。
- ViT在59.8%计算量减少下仍达83.41%准确率，速度提升1.388倍。
- 适合追求高效推理的模型部署者，尤其关注轻量化与快速验证。
本文提出一种基于马尔琴科-帕斯图（Marchenko--Pastur）随机矩阵理论的深度神经网络剪枝方法，可在极短的微调周期内保持高精度。理论提供确定性数据路径证书：若移除部分对输出逻辑值影响小，则剪枝可降低弹性网络目标并保留原始边际足够大的样本。零预算剪枝实现完美剪枝；剪枝-恢复扩展模型在固定稀疏执行模式下模拟权重恢复；添加L2正则化后，随机成分在训练极限下消失，而稳定峰值持续存在。在独立同分布高斯条件下，拟合的马尔琴科-帕斯图边界σ+提供层级剪枝预算的高概率信号。ImageNet-1k实验中，仅经三轮蒸馏，ViT-B/16 2:4+ToMe达到83.41% top-1准确率（较密集模型低1.70个百分点），计算量减少至59.81%，在相同检查点和ToMe图下，A40原生2:4后端加速1.388倍；另一无ToMe A100端点达2.705倍加速。结构化稀疏下，ViT-B/16 6:12达83.74%，ViT-L/16 8:16密度+置换达85.33%（-0.51 pp），ConvNeXtV2-Base 12:16达86.35%（-0.37 pp）。CNN方面，ResNet50 8:16密度+置换达75.87%（-0.26 pp），ResNet152d CAST-conv+置换达81.33%（-1.53 pp），计算量约50%，在A40 im2col+2:4稀疏GEMM审计下提速1.62倍。
原文摘要 · Abstract (English)
We study a Marchenko--Pastur (MP) random-matrix approach to pruning deep neural networks with very small post-pruning fine-tuning budgets. The main practical contribution is accuracy retention under short calibration and fine-tuning schedules, rather than a long post-pruning reoptimization pipeline. The theory gives deterministic data-path certificates: if the removed component $R$ has small propagated logit effect $L_s \| R ψ_1(s) \|_\infty$, pruning decreases an elastic-net objective and preserves samples whose dense margin exceeds twice the perturbation. The zero-budget case gives perfect pruning; a prune--restore extension models weight restoration inside a fixed sparse-execution pattern; and an additive $L_2$-regularized model shows admissible random-like components vanish at the training limit, with persistent spikes stabilizing as the MP bulk collapses. Under iid-Gaussian sufficient conditions, the fitted MP edge $σ_+$ gives a high-probability layerwise budget signal. On ImageNet-1k, after only three distillation epochs, ViT-B/16 $2{:}4{+}$ToMe reaches $83.41\%$ top-1 ($-1.70$ pp from dense) at $59.81\%$ sparse-execution MAC reduction, with $1.388\times$ best-observed A40 native-$2{:}4$ backend speedup for the same checkpoint and ToMe graph; a separate no-ToMe A100 endpoint gives $2.705\times$. At structured sparsity, ViT-B/16 $6{:}12$ reaches $83.74\%$, ViT-L/16 $8{:}16$ dense+permutation reaches $85.33\%$ ($-0.51$ pp), and ConvNeXtV2-Base $12{:}16$ reaches $86.35\%$ ($-0.37$ pp). For CNNs, ResNet50 $8{:}16$ dense+permutation reaches $75.87\%$ ($-0.26$ pp), and ResNet152d CAST-conv+permutation reaches $81.33\%$ ($-1.53$ pp) at ${\sim}50\%$ MAC accounting with a $1.62\times$ A40 im2col$+2{:}4$ sparse-GEMM audit.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。