动态稀疏训练比密集训练更抗图像损坏，且不增加资源开销。
Dynamic Sparse Training versus Dense Training: The Unexpected Winner in Image Corruption Robustness
- 在10%至50%稀疏度下，动态稀疏训练提升模型鲁棒性。
- 在图像与视频数据上，稀疏训练在多种架构和算法中均优于密集训练。
- 适合追求高鲁棒性且不以极致效率为目标的深度学习研究者。
通常认为，动态稀疏训练虽提升可扩展性与效率，但会牺牲分类精度；而密集训练被视为最大化模型对图像损坏鲁棒性的标准方法。本文挑战这一共识：在非强调效率的前提下（稀疏度10%至50%），动态稀疏训练在图像与视频数据上，使用多种经典与现代视觉模型及三种主流动态稀疏训练算法，均持续优于密集训练，且不增加或甚至降低资源成本。研究揭示了动态稀疏训练未被认知的新优势，为突破现有深度学习鲁棒性极限提供了新路径。
原文摘要 · Abstract (English)
It is generally perceived that Dynamic Sparse Training opens the door to a new era of scalability and efficiency for artificial neural networks at, perhaps, some costs in accuracy performance for the classification task. At the same time, Dense Training is widely accepted as being the "de facto" approach to train artificial neural networks if one would like to maximize their robustness against image corruption. In this paper, we question this general practice. Consequently, we claim that, contrary to what is commonly thought, the Dynamic Sparse Training methods can consistently outperform Dense Training in terms of robustness accuracy, particularly if the efficiency aspect is not considered as a main objective (i.e., sparsity levels between 10% and up to 50%), without adding (or even reducing) resource cost. We validate our claim on two types of data, images and videos, using several traditional and modern deep learning architectures for computer vision and three widely studied Dynamic Sparse Training algorithms. Our findings reveal a new yet-unknown benefit of Dynamic Sparse Training and open new possibilities in improving deep learning robustness beyond the current state of the art.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。