ResNet通过最小化复杂度实现高效计算，揭示深度学习的简洁性优势。
Deep Learning as a Convex Paradigm of Computation: Minimizing Circuit Size with ResNets
- 用残差网络参数定义函数复杂度，等价于最小化电路节点数。
- 在特定条件下，最小化该复杂度可逼近最优电路规模（差2的幂次）。
- 为深度学习成功提供新解释，适合理解模型简洁性的研究者。
本文提出深度神经网络实现了计算中的奥卡姆剃刀原则——寻找最简单的拟合数据算法，并可能解释其在众多任务中超越传统统计方法的卓越表现。研究发现，在‘比蒙特卡罗更难’（HTMC）背景下，当γ > 2时，能被大小不超过cε⁻ᵞ的二值电路ε-近似的实值函数集合呈现凸性，从而可在函数空间上定义HTMC范数。同时，对残差网络（ResNets）参数可定义加权ℓ₁范数，进而诱导出函数层面的‘ResNet范数’。这两个范数之间存在近乎匹配的夹逼不等式关系。因此，最小化该ResNet范数等价于寻找一个与最优电路规模仅相差2的幂次的电路结构。由此，ResNets被视为一种更适合HTMC背景、具有凸性的新型实函数计算模型。
原文摘要 · Abstract (English)
This paper argues that DNNs implement a computational Occam's razor -- finding the `simplest' algorithm that fits the data -- and that this could explain their incredible and wide-ranging success over more traditional statistical methods. We start with the discovery that the set of real-valued function $f$ that can be $ε$-approximated with a binary circuit of size at most $cε^{-γ}$ becomes convex in the `Harder than Monte Carlo' (HTMC) regime, when $γ>2$, allowing for the definition of a HTMC norm on functions. In parallel one can define a complexity measure on the parameters of a ResNets (a weighted $\ell_1$ norm of the parameters), which induce a `ResNet norm' on functions. The HTMC and ResNet norms can then be related by an almost matching sandwich bound. Thus minimizing this ResNet norm is equivalent to finding a circuit that fits the data with an almost minimal number of nodes (within a power of 2 of being optimal). ResNets thus appear as an alternative model for computation of real functions, better adapted to the HTMC regime and its convexity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。