arXiv:2502.05073math.PRcs.CC2025-02被引 2

揭示分层函数的噪声敏感性,证明其学习难度随深度指数增长。

Noise Sensitivity and Learning Lower Bounds for Hierarchical Functions

  • 分析树状分层函数在独立输入下的噪声稳定性
  • 深度越大,噪声稳定性呈指数下降,学习难度剧增
  • 适用于理解神经网络、统计查询学习的下限

近期研究通过考察具有分层结构的函数或数据来解析深度学习的成功机制。为研究分层函数的学习复杂度,本文分析了在独立输入下具有树状分层结构函数的噪声稳定性。若层级中每个函数与线性函数的距离至少为ε,那么其噪声稳定性随层级深度呈指数衰减。该结果在非适应性学习中有直接应用:基于Dachman-Soled, Feldman, Tan, Wan and Wimmer (2014) 的成果,在布尔设置下,可推出对基于分层函数类的统计查询(SQ)学习存在超多项式下界。此外,我们还基于临界位点渗流中的穿越事件指示函数导出类似下界,尽管这些事件不完全符合定义中的分层结构,但具备数学物理中研究的分层特征。结合Abbe et al. (2022) 的结果,该工作意味着在全连接神经网络上用梯度下降学习分层函数所需的样本复杂度也存在下界。最后,在高斯设置下,利用Diakonikolas, Kane, Pittas and Zarifis (2021) 的成果,进一步给出超多项式下界用于非适应性统计查询学习。

原文摘要 · Abstract (English)

Recent works explore deep learning's success by examining functions or data with hierarchical structure. To study the learning complexity of functions with hierarchical structure, we study the noise stability of functions with tree hierarchical structure on independent inputs. We show that if each function in the hierarchy is $\varepsilon$-far from linear, the noise stability is exponentially small in the depth of the hierarchy. Our results have immediate applications for agnostic learning. In the Boolean setting using the results of Dachman-Soled, Feldman, Tan, Wan and Wimmer (2014), our results provide Statistical Query super-polynomial lower bounds for agnostically learning classes that are based on hierarchical functions. We also derive similar SQ lower bounds based on the indicators of crossing events in critical site percolation. These crossing events are not formally hierarchical as we define but still have some hierarchical features as studied in mathematical physics. Using the results of Abbe, Bengio, Cornacchiam, Kleinberg, Lotfi, Raghu and Zhang (2022), our results imply sample complexity lower bounds for learning hierarchical functions with gradient descent on fully connected neural networks. Finally in the Gaussian setting, using the results of Diakonikolas, Kane, Pittas and Zarifis (2021), our results provide super-polynomial lower bounds for agnostic SQ learning.

分层函数学习下界噪声稳定性统计查询

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。