神经网络优化中，维度越高，收敛越慢，与函数光滑性密切相关。
Curse of Dimensionality in Neural Network Optimization
- 用2-Wasserstein流分析参数分布演化，揭示优化动态。
- 在d维空间中，风险下降速度不超过t^(-4r/(d-2r))。
- 即使使用局部Lipschitz激活函数，维度灾难仍存在。
本文证明，当使用Lipschitz连续激活函数的浅层神经网络，通过经验或总体风险最小化来逼近[0,1]^d上r次连续可微的目标函数时，总体风险的衰减速率不会快于t^(-4r/(d-2r))，其中t为梯度流动力学的时间参数。该结果凸显了优化所需精度下的维度灾难现象。训练动态通过参数分布的2-Wasserstein梯度流演化进行分析，而非直接追踪参数变化。进一步表明，当采用局部Lipschitz连续激活函数（在[-x,x]上Lipschitz常数有界于O(x^δ)）时，总体风险衰减速度仍不快于t^(-((4+2δ)r)/(d-2r))。本工作旨在揭示函数光滑性如何影响神经网络优化中的维度灾难，这是当前理论研究中一个重要但未充分探索的方向。
原文摘要 · Abstract (English)
This paper demonstrates that when a shallow neural network with a Lipschitz continuous activation function is trained using either empirical or population risk to approximate a target function that is $r$ times continuously differentiable on $[0,1]^d$, the population risk may not decay at a rate faster than $t^{-\frac{4r}{d-2r}}$, where $t$ denotes the time parameter of the gradient flow dynamics. This result highlights the presence of the curse of dimensionality in the optimization computation required to achieve a desired accuracy. Instead of analyzing parameter evolution directly, the training dynamics are examined through the evolution of the parameter distribution under the 2-Wasserstein gradient flow. Furthermore, it is established that the curse of dimensionality persists when a locally Lipschitz continuous activation function is employed, where the Lipschitz constant in $[-x,x]$ is bounded by $O(x^δ)$ for any $x \in \mathbb{R}$. In this scenario, the population risk is shown to decay at a rate no faster than $t^{-\frac{(4+2δ)r}{d-2r}}$. Understanding how function smoothness influences the curse of dimensionality in neural network optimization theory is an important and underexplored direction that this work aims to address.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。