揭示了ReLU层的单射容量与球形感知机的深层关联
Injectivity capacity of ReLU gates
- 通过提升随机对偶理论解析ReLU层单射性
- 三层提升后估计误差小于0.1%，收敛极快
- 发现参数间显式关系，匹配经典复现预测
本文研究ReLU网络层的单射性质。确定ReLU层的单射容量（输入与输出数量之比）等价于求解所谓的ℓ₀球形感知机问题。借助全提升随机对偶理论（fl RDT），发展出一套强大工具用于处理ℓ₀球形感知机，并间接解决ReLU层的单射性问题。为使fl RDT实际可用，开展了大规模数值实验。结果表明，提升机制在第三层即实现惊人快速收敛，估计量相对误差不超过∼0.1%。同时，揭示了关键提升参数间的闭式解析关系。这些关系不仅极大简化了数值计算，还揭示了提升结构内部精妙的参数关联。最终结果与文献[40]的副本预测高度吻合。
原文摘要 · Abstract (English)
We consider the injectivity property of the ReLU networks layers. Determining the ReLU injectivity capacity (ratio of the number of layer's inputs and outputs) is established as isomorphic to determining the capacity of the so-called $\ell_0$ spherical perceptron. Employing \emph{fully lifted random duality theory} (fl RDT) a powerful program is developed and utilized to handle the $\ell_0$ spherical perceptron and implicitly the ReLU layers injectivity. To put the entire fl RDT machinery in practical use, a sizeable set of numerical evaluations is conducted as well. The lifting mechanism is observed to converge remarkably fast with relative corrections in the estimated quantities not exceeding $\sim 0.1\%$ already on the third level of lifting. Closed form explicit analytical relations among key lifting parameters are uncovered as well. In addition to being of incredible importance in handling all the required numerical work, these relations also shed a new light on beautiful parametric interconnections within the lifting structure. Finally, the obtained results are also shown to fairly closely match the replica predictions from [40].
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。