研究了带理性权重的ReLU网络深度下界，发现其深度随输入维度增长而增加。
On the Expressiveness of Rational ReLU Neural Networks With Bounded Depth
- 在有理数权重约束下，表示最大函数需至少⌈log₃(n+1)⌉层
- 当权重为N进制分数时，深度下界为Ω((ln n)/(ln ln N))
- 首次给出实用型ReLU网络的非平凡深度下界，适合理论学习者
为验证ReLU神经网络表达能力随深度增强，文献中常考察函数F_n = max{0, x₁, ..., xₙ}。Hertrich、Basu、Di Summa与Skutella（NeurIPS 2021）提出猜想：任何精确表示F_n的ReLU网络至少需⌈log₂(n+1)⌉个隐藏层。该猜想已被Haase、Hertrich与Loho（ICLR 2023）在整数权重情况下证实。本文延续此研究，证明在权重为十进制小数的ReLU网络中，表示F_n至少需要⌈log₃(n+1)⌉层；若所有权重为N进制分数，则至少需要Ω((ln n)/(ln ln N))层。这些结果部分验证了原猜想在有理数权重下的正确性，并首次给出了实际相关ReLU网络的非恒定深度下界。
原文摘要 · Abstract (English)
To confirm that the expressive power of ReLU neural networks grows with their depth, the function $F_n = \max \{0,x_1,\ldots,x_n\}$ has been considered in the literature. A conjecture by Hertrich, Basu, Di Summa, and Skutella [NeurIPS 2021] states that any ReLU network that exactly represents $F_n$ has at least $\lceil\log_2 (n+1)\rceil$ hidden layers. The conjecture has recently been confirmed for networks with integer weights by Haase, Hertrich, and Loho [ICLR 2023]. We follow up on this line of research and show that, within ReLU networks whose weights are decimal fractions, $F_n$ can only be represented by networks with at least $\lceil\log_3 (n+1)\rceil$ hidden layers. Moreover, if all weights are $N$-ary fractions, then $F_n$ can only be represented by networks with at least $Ω( \frac{\ln n}{\ln \ln N})$ layers. These results are a partial confirmation of the above conjecture for rational ReLU networks, and provide the first non-constant lower bound on the depth of practically relevant ReLU networks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。