揭示三值神经网络线性区域增长规律，解释其实际表现优异的原因。
A Lower Bound for the Number of Linear Regions of Ternary ReLU Regression Neural Networks
- 从线性区域数量角度分析三值ReLU网络的表达能力。
- 网络宽度每翻倍或深度翻倍，线性区域数呈多项式与指数增长。
- 理论证明三值网络可达到与普通网络相当的表达上限，适合关注模型压缩者。
随着深度学习发展，降低计算复杂度和内存消耗成为关键挑战，将参数限制在{-1, 0, +1}的三值神经网络（Ternary NNs）因其潜力受到关注。尽管三值网络在图像识别、自然语言处理等任务中表现优异，但其理论理解仍不充分。本文从线性区域数量角度，理论分析了使用ReLU激活函数的三值回归神经网络的表达能力。研究证明，其线性区域数量随网络宽度呈多项式增长，随深度呈指数增长，与标准神经网络类似。进一步表明，只需将三值网络宽度加倍，再通过平方宽度或翻倍深度，即可获得与通用ReLU回归网络相当的最大线性区域数下界；若所有层均使用ReLU，还需额外加倍宽度。该结果从理论上部分解释了三值网络在实践中成功的原因。
原文摘要 · Abstract (English)
With the advancement of deep learning, reducing computational complexity and memory consumption has become a critical challenge, and ternary neural networks (NNs) that restrict parameters to $\{-1, 0, +1\}$ have attracted attention as a promising approach. While ternary NNs demonstrate excellent performance in practical applications such as image recognition and natural language processing, their theoretical understanding remains insufficient. In this paper, we theoretically analyze the expressivity of ternary NNs from the perspective of the number of linear regions. Specifically, we evaluate the number of linear regions of ternary regression NNs with Rectified Linear Unit (ReLU) for activation functions and prove that the number of linear regions increases polynomially with respect to network width and exponentially with respect to depth, similar to standard NNs. Moreover, we show that it suffices to first double the width, then either square the width or double the depth of ternary NNs with alternating ReLU and identity layers to achieve a lower bound on the maximum number of linear regions comparable to that of general ReLU regression NNs. When using ReLU in all the layers, a similar bound is obtained by further doubling the width. This provides a theoretical explanation, in some sense, for the practical success of ternary NNs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。