通过动态剪掉不活跃层，训练速度提升超50%且精度损失小。
Learn&Drop: Fast Learning of CNNs based on Layer Dropping
- 训练时评估各层参数变化，自动剪掉停止学习的层
- 前向传播计算量减少17.83%至83.74%，训练时间减半以上
- 适合需要持续微调或在线学习的场景
本文提出一种新方法，通过在训练过程中评估每层参数变化程度，判断其是否仍在学习，并据此动态裁剪网络。该方法聚焦于降低训练时前向传播的计算开销，而非推理压缩或反向传播操作限制。在VGG和ResNet两大主流架构上验证，于MNIST、CIFAR-10和Imagenette数据集上实验表明，模型训练时间显著缩短超过50%，准确率基本不变。前向传播的FLOPs减少幅度从VGG-11的17.83%到ResNet-152的83.74%不等，证明该技术可高效加速卷积神经网络的学习过程，尤其适用于数据流式到达下的微调或在线训练任务。
原文摘要 · Abstract (English)
This paper proposes a new method to improve the training efficiency of deep convolutional neural networks. During training, the method evaluates scores to measure how much each layer's parameters change and whether the layer will continue learning or not. Based on these scores, the network is scaled down such that the number of parameters to be learned is reduced, yielding a speed up in training. Unlike state-of-the-art methods that try to compress the network to be used in the inference phase or to limit the number of operations performed in the backpropagation phase, the proposed method is novel in that it focuses on reducing the number of operations performed by the network in the forward propagation during training. The proposed training strategy has been validated on two widely used architecture families: VGG and ResNet. Experiments on MNIST, CIFAR-10 and Imagenette show that, with the proposed method, the training time of the models is more than halved without significantly impacting accuracy. The FLOPs reduction in the forward propagation during training ranges from 17.83\% for VGG-11 to 83.74\% for ResNet-152. These results demonstrate the effectiveness of the proposed technique in speeding up learning of CNNs. The technique will be especially useful in applications where fine-tuning or online training of convolutional models is required, for instance because data arrive sequentially.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。