提出新型结构化稀疏训练方法，实现高速推理与训练。
Dynamic Sparse Training of Diagonally Sparse Networks
- 采用对角线稀疏模式，全程保持稀疏计算。
- 在ViT中90%层稀疏时，推理速度提升3.13倍，训练提速1.59倍。
- 适配GPU硬件，适合追求高效部署的模型研发者。
动态稀疏训练（DST）在结构化和非结构化稀疏场景中取得了进展，能在大幅减少参数量的同时逼近稠密模型性能，促进模型扩展。然而，非结构化稀疏难以在现代硬件上带来实际加速。为此，我们提出DynaDiag，一种新型的结构化到结构化的动态稀疏训练方法，性能媲美非结构化稀疏。DynaDiag在整个训练过程中强制执行对角线稀疏模式，并在前向与反向传播中保持稀疏计算。我们进一步利用对角线结构，通过自定义CUDA内核加速计算，实现硬件友好性。在多种神经网络架构上的实证评估表明，该方法在保持与非结构化方法相当精度的同时，获得了显著的计算收益。特别地，在90%稀疏的ViT线性层中，线上推理速度最高提升3.13倍，且不损失模型性能；在GPU上训练速度相比等效非结构化层提升1.59倍。代码已开源：https://github.com/horizon-research/DynaDiag/
原文摘要 · Abstract (English)
Recent advances in Dynamic Sparse Training (DST) have pushed the frontier of sparse neural network training in structured and unstructured contexts, matching dense-model performance while drastically reducing parameter counts to facilitate model scaling. However, unstructured sparsity often fails to translate into practical speedups on modern hardware. To address this shortcoming, we propose DynaDiag, a novel structured sparse-to-sparse DST method that performs at par with unstructured sparsity. DynaDiag enforces a diagonal sparsity pattern throughout training and preserves sparse computation in forward and backward passes. We further leverage the diagonal structure to accelerate computation via a custom CUDA kernel, rendering the method hardware-friendly. Empirical evaluations on diverse neural architectures demonstrate that our method maintains accuracy on par with unstructured counterparts while benefiting from tangible computational gains. Notably, with 90% sparse linear layers in ViTs, we observe up to a 3.13x speedup in online inference without sacrificing model performance and a 1.59x speedup in training on a GPU compared to equivalent unstructured layers. Our source code is available at https://github.com/horizon-research/DynaDiag/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。