提出结构化丢弃方法，让GPU加速训练同时保持良好泛化性能。
Efficient Sparse Training with Structured Dropout
- 采用结构化丢弃机制，适配GPU硬件加速
- 低稀疏度下仍实现比密集模型更快的训练速度
- 可直接替换标准丢弃，适合追求训练效率的研究者
Dropout是深度学习中常用的正则化技术，能提升模型泛化能力。尽管其引入稀疏性，理论上可提高吞吐量，但由于非结构化特性，通常无法在GPU上带来实际加速。本文提出SparseDrop，一种结构化且硬件友好的丢弃变体，能够有效利用稀疏性。我们提供了SparseDrop的CUDA实现，在低稀疏度下即实现了对密集版本的加速。实验表明，SparseDrop在正则化效果上与标准dropout相当，甚至在某些情况下更优。这表明它有望作为标准dropout的即插即用替代方案，实现更快的训练速度。代码已开源：https://github.com/andylolu2/sparse-dropout。
原文摘要 · Abstract (English)
Dropout is a common regularisation technique in deep learning that improves generalisation. Even though it introduces sparsity and thus potential for higher throughput, it usually cannot bring speed-ups on GPUs due to its unstructured nature. In this project, I experiment with SparseDrop, a structured, hardware-friendly variant of dropout that can exploit such sparsity. I provide a CUDA implementation of SparseDrop, achieving speed-ups against its dense counterpart even at low sparsity levels. The empirical results demonstrate that SparseDrop provides similar, or sometimes even better, regularisation properties as standard dropout. This suggests its potential as a drop-in replacement to standard dropout with faster training speeds. The source code is available at https://github.com/andylolu2/sparse-dropout
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。