揭示深度ReLU网络的运行机制，解释训练结果为何如此。
On the Principles of Deep Feedforward ReLU Networks
- 用路径关系分析多层ReLU网络的分段线性结构。
- 深层网络可形成分段线性流形，而非两层网络的超平面。
- 原理可推广至复杂训练解，适合研究神经网络机理者阅读。
深度前馈神经网络架构在深度学习中广泛应用，无论是作为完整系统还是其他架构的子网络，其工作机制是理解神经网络黑箱的关键。本文基于最简单的两层ReLU网络，系统研究了具有多个隐藏层的深度前馈ReLU网络，成功解释了反向传播算法获得的训练解。路径概念，尤其是路径间的相互关系，在揭示黑箱之谜中起核心作用。研究表明，深层ReLU网络中的单元可形成分段线性流形以划分输入空间，而非两层情况下的超平面。如何高效利用隐藏层单元同时生成线性函数与输入空间的分割，也是一个核心问题。两层ReLU网络的基本原理在很大程度上可推广至深层情形，例如多重严格偏序和连续性约束。这些基础而简单的原则组合后可产生复杂的实例化结果，包括训练解，从这个意义上说，深度前馈ReLU网络的黑箱得以揭示。
原文摘要 · Abstract (English)
The architecture of deep feedforward neural networks is ubiquitous in deep learning, either as a whole system or as a subnetwork of other architectures, and thus its mechanism is a key ingredient of the black box of neural networks. On the basis of the simplest two-layer ReLU network, this paper systematically studies the mechanism of deep feedforward ReLU networks with multiple hidden layers and successfully explains the training solution obtained by the back-propagation algorithm. The concept of a path, especially in terms of the relationships between paths, plays a central role in uncovering the mystery of the black box. It is shown that a unit of a deep ReLU network can form a piecewise linear manifold to divide the input space, instead of a hyperplane of the two-layer case. How to efficiently use the hidden-layer units to produce both linear functions and partitions of the input space is also a central problem. The principles of a two-layer ReLU network can be generalized to the deeper case to a large extent, such as multiple strict partial orders and continuity restriction. The combination of the basic and simple principles proposed can yield complicated instantiations including the training solutions, and in this sense the black box of deep feedforward ReLU networks is revealed.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。