通过反向传播随机丢弃层，加速深度网络训练且不改变模型结构。
LayerDropBack: A Universally Applicable Approach for Accelerating Training of Deep Networks
- 仅在反向传播时随机丢弃层，保持前向传播一致。
- 在多个模型上实现16.93%~23.97%的训练时间减少。
- 无需修改架构，适用于ViT、Swin Transformer等各类网络。
训练超深卷积网络耗时且需大量计算资源。现有加速方法常依赖特定架构或需修改网络结构。我们提出LayerDropBack（LDB），一种简单高效的通用加速方法，仅在反向传播中引入随机性，保持前向传播完整，确保训练与推理使用同一网络。LDB可无缝集成到任意模型训练流程中，无需修改架构，适配多种网络拓扑。在ViT、Swin Transformer、EfficientNet、DLA等多种架构及CIFAR-100、ImageNet数据集上的实验表明，训练时间显著缩短16.93%至23.97%，同时保持或提升模型精度。代码已公开于https://github.com/neiterman21/LDB。
原文摘要 · Abstract (English)
Training very deep convolutional networks is challenging, requiring significant computational resources and time. Existing acceleration methods often depend on specific architectures or require network modifications. We introduce LayerDropBack (LDB), a simple yet effective method to accelerate training across a wide range of deep networks. LDB introduces randomness only in the backward pass, maintaining the integrity of the forward pass, guaranteeing that the same network is used during both training and inference. LDB can be seamlessly integrated into the training process of any model without altering its architecture, making it suitable for various network topologies. Our extensive experiments across multiple architectures (ViT, Swin Transformer, EfficientNet, DLA) and datasets (CIFAR-100, ImageNet) show significant training time reductions of 16.93\% to 23.97\%, while preserving or even enhancing model accuracy. Code is available at \url{https://github.com/neiterman21/LDB}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。