用DCA框架解释残差连接为何有效，还提出新模型NegNet。
Understand the Effectiveness of Shortcuts through the Lens of DCA
- 将残差网络梯度视为对普通网络的DCA优化结果。
- 新架构NegNet性能媲美ResNet，且可纳入DCA框架。
- 为理解深度网络中的捷径机制提供统一视角。
差分凸算法（DCA）是一种经典的非凸优化方法，适用于可表示为两个凸函数之差的非凸函数最小化问题。许多知名优化算法，如SGD和近端点法，均可视为特定DC分解下的DCA，使其成为强大的优化框架。另一方面，捷径是现代深度神经网络的关键结构特征，有助于训练与优化。本文表明，残差神经网络的梯度可通过将DCA应用于无捷径的原始网络得到。因此，从DCA视角可更深入理解捷径网络的有效性。此外，我们提出一种新架构NegNet，其不符合传统解释，但性能与ResNet相当，并可被纳入DCA框架。
原文摘要 · Abstract (English)
Difference-of-Convex Algorithm (DCA) is a well-known nonconvex optimization algorithm for minimizing a nonconvex function that can be expressed as the difference of two convex ones. Many famous existing optimization algorithms, such as SGD and proximal point methods, can be viewed as special DCAs with specific DC decompositions, making it a powerful framework for optimization. On the other hand, shortcuts are a key architectural feature in modern deep neural networks, facilitating both training and optimization. We showed that the shortcut neural network gradient can be obtained by applying DCA to vanilla neural networks, networks without shortcut connections. Therefore, from the perspective of DCA, we can better understand the effectiveness of networks with shortcuts. Moreover, we proposed a new architecture called NegNet that does not fit the previous interpretation but performs on par with ResNet and can be included in the DCA framework.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。