arXiv:2503.12744cs.LGcs.IT2025-03

研究有限样本下浅层神经网络的精确识别能力,区分不同激活函数的表现。

Finite Samples for Shallow Neural Networks

  • 通过构造采样点,验证不同激活函数下网络可被有限样本唯一确定
  • ReLU网络无法用有限样本完全识别,而logistic与tanh可实现精确恢复
  • 首次揭示激活函数对小样本条件下网络可识别性的影响

本文研究有限样本对带不同非线性激活函数(包括ReLU、逻辑斯蒂sigmoid和双曲正切)的两层不可约浅层神经网络的识别能力。不可约网络指其函数无法由更少神经元的网络表示。针对ReLU激活,我们建立了网络不可约性的充要条件,并证明:有限样本不足以对任意不可约ReLU浅层网络做出确定性识别。然而,对于给定的不可约网络,可构造有限采样点集将其与其他同规模网络区分开。相反,对于logistic sigmoid和tanh等解析激活函数,我们给出了正面结果:可构造有限样本实现两层不可约浅层解析网络的恢复。据我们所知,这是首个基于有限样本函数值研究两层不可约网络精确识别的论文。研究结果为不同激活函数在有限采样下的性能比较提供了理论依据。

原文摘要 · Abstract (English)

This paper investigates the ability of finite samples to identify two-layer irreducible shallow networks with various nonlinear activation functions, including rectified linear units (ReLU) and analytic functions such as the logistic sigmoid and hyperbolic tangent. An ``irreducible" network is one whose function cannot be represented by another network with fewer neurons. For ReLU activation functions, we first establish necessary and sufficient conditions for determining the irreducibility of a network. Subsequently, we prove a negative result: finite samples are insufficient for definitive identification of any irreducible ReLU shallow network. Nevertheless, we demonstrate that for a given irreducible network, one can construct a finite set of sampling points that can distinguish it from other network with the same neuron count. Conversely, for logistic sigmoid and hyperbolic tangent activation functions, we provide a positive result. We construct finite samples that enable the recovery of two-layer irreducible shallow analytic networks. To the best of our knowledge, this is the first study to investigate the exact identification of two-layer irreducible networks using finite sample function values. Our findings provide insights into the comparative performance of networks with different activation functions under limited sampling conditions.

神经网络激活函数有限样本可识别性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。