用高阶差分方法改进残差网络,不增加模型大小就能提升性能。
Enhancing Convolutional Neural Networks with Higher-Order Numerical Difference Methods
- 基于线性多步法设计新堆叠结构,替代传统低阶差分方法。
- 在ResNet上实现更优准确率,且模型规模不变。
- 方法可推广至其他神经网络,理论基础扎实。
随着深度学习在实际应用中的兴起,卷积神经网络(CNN)已能协助人类解决诸多现实问题。为提升CNN性能,研究者探索了多种网络架构,部分基于长期经验,部分通过神经架构搜索生成。然而,这些方法受限于模型大小与环境约束,难以完全发挥性能优势。近年来研究发现,许多CNN结构可解释为常微分方程的离散化形式,暗示可通过高阶数值差分方法设计具有理论支持的深层网络。现有多数CNN基于低阶数值方法,而线性多步法精度高于前向欧拉法。本文提出一种基于线性多步法的堆叠方案,在不增加模型大小的前提下增强ResNet性能,并与龙格-库塔方案对比。实验表明,该方案性能优于现有堆叠结构(ResNet与HO-ResNet),且具备拓展至其他神经网络的能力。
原文摘要 · Abstract (English)
With the rise of deep learning technology in practical applications, Convolutional Neural Networks (CNNs) have been able to assist humans in solving many real-world problems. To enhance the performance of CNNs, numerous network architectures have been explored. Some of these architectures are designed based on the accumulated experience of researchers over time, while others are designed through neural architecture search methods. The improvements made to CNNs by the aforementioned methods are quite significant, but most of the improvement methods are limited in reality by model size and environmental constraints, making it difficult to fully realize the improved performance. In recent years, research has found that many CNN structures can be explained by the discretization of ordinary differential equations. This implies that we can design theoretically supported deep network structures using higher-order numerical difference methods. It should be noted that most of the previous CNN model structures are based on low-order numerical methods. Therefore, considering that the accuracy of linear multi-step numerical difference methods is higher than that of the forward Euler method, this paper proposes a stacking scheme based on the linear multi-step method. This scheme enhances the performance of ResNet without increasing the model size and compares it with the Runge-Kutta scheme. The experimental results show that the performance of the stacking scheme proposed in this paper is superior to existing stacking schemes (ResNet and HO-ResNet), and it has the capability to be extended to other types of neural networks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。