通过学习排列矩阵，让结构化稀疏模型逼近无结构稀疏的精度表现。
SHUFFLESPARSE: Learned Shuffles for Structured Sparse Networks
- 引入统一的可学习排列操作，提升结构化稀疏的表达能力。
- 在90%-95%稀疏度下，ViT和GPT-2精度差距缩小至接近无结构稀疏。
- 适用于多种结构（块、N:M、对角线），且推理开销低于8.7%。
结构化权重稀疏能在现代GPU上加速训练与推理，但在极端稀疏度下准确率仍落后于无结构动态稀疏训练（DST）。我们发现其根本原因在于表达能力不足：密集层可实现任意非零权重模式，而结构化模式仅限于少数配置。为此提出SHUFFLESPARSE，一种跨不同结构（如块、N:M、对角线）和方法（从零开始训练、一次性剪枝）通用的单个排列算子。该方法联合学习一个排列矩阵与结构化权重矩阵，显著缩小结构化与无结构稀疏的精度差距。在ViT-B16（ImageNet-1K）和GPT-2（WikiText-103）上，90%-95%稀疏度下性能接近无结构水平，推理开销小于8.7%，同时保持原有训练加速优势。该方法亦适用于预训练大模型的一次性2:4剪枝，在LLaMA-2 7B上零样本准确率提升4.6点。结果表明，学习排列是恢复结构化稀疏准确率的有效通用策略。
原文摘要 · Abstract (English)
Structured weight sparsity accelerates training and inference on modern GPUs, but it trails unstructured dynamic sparse training (DST) in accuracy especially at extreme sparsity. We pinpoint the reason for this difference in performance to a lack of expressivity: a dense layer can implement any pattern of non-zero weights, whereas structured patterns are restricted to only a small set of weight configurations. We introduce SHUFFLESPARSE, a single permutation primitive that applies uniformly across DST-from-scratch and one-shot pruning, and across N:M, block etc. We close most of this gap by learning a single permutation matrix jointly with the structured weight matrix. When used on three different types of structures (block, N:M, and diagonal), SHUFFLESPARSE is able to reduce the structured-vs-unstructured accuracy gap on ViT-B16 (ImageNet-1K) and GPT-2 (WikiText-103) at 90-95% sparsity, while adding minimal inference overhead (< 8.7% inference overhead) and preserving any training acceleration the host structure provides. The same permutation formulation transfers to one-shot 2:4 pruning of pretrained LLMs, where it improves zero-shot accuracy by 4.6 points on LLaMA-2 7B. Together, these results establish learned permutations as a general tool for recovering unstructured-level accuracy from structured sparsity patterns.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。