有限精度下tanh网络学习受限于蒙特卡洛速率
Limitations of Learning Tanh Neural Networks with Finite Precision
- 用迭代tanh构造局域波峰函数，揭示学习机制
- 采样数为m时，收敛率不超过O(m^{-1/p})
- 适用于研究神经网络精度与泛化能力的学者
我们研究在有限精度计算和L^p精度保证下，从点采样学习tanh神经网络的局限性，基于Berner、Grohs和Voigtländer（2023）的工作。通过一种新型的迭代tanh激活构造的局域波峰函数，我们证明：在有限精度下，任何基于m个样本的自适应随机算法，在L^p范数中无法获得高于蒙特卡洛速率O(m^{-1/p})的收敛率，除非采样预算随网络参数规模和结构呈指数增长。结果揭示了有限精度对包含局域波峰函数类学习能力的根本限制，将此前针对ReLU网络的结果扩展至tanh情形。
原文摘要 · Abstract (English)
We investigate limitations of learning $\tanh$ neural networks from point evaluations under finite-precision computations and $L^p$ accuracy guarantees, building on Berner, Grohs, and Voigtländer (2023). Our approach is based on a novel construction of sharply localized bump functions via iterated $\tanh$ activations. Using this mechanism, we show that, in a finite-precision setting, no adaptive randomized algorithm based on $m$ samples can achieve a convergence rate higher than the Monte Carlo rate $O(m^{-1/p})$ in the $L^p$ norm, unless the sampling budget grows exponentially with the size of the network parameters and architecture. The results reveal fundamental limitations imposed by finite precision on the learnability of classes containing localized bump functions, extending previous results for ReLU networks to the $\tanh$ setting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。