梯度下降可视为频谱偏置的隐式调控器，影响模型保留的频率成分。
Gradient Descent as a Shrinkage Operator for Spectral Bias
- 将梯度下降重理解为对雅可比矩阵奇异值的收缩操作。
- 学习率与迭代次数决定保留的频率分量数量（带宽）。
- 单调激活函数下梯度下降具正则化效果，非单调函数更高效
我们拓展了激活函数与样条回归/平滑之间的联系，刻画其在1维浅层网络中对频谱偏置的影响。进一步表明，梯度下降（GD）可被重新解释为一种收缩算子，作用于神经网络雅可比矩阵的奇异值。从这一视角看，GD隐式选择需保留的频率成分数量，从而控制频谱偏置。本文提出学习率与迭代次数和带宽（活跃成分数）间的显式关系。结果表明，仅当使用单调激活函数时，梯度下降才具备正则化效果。最后，我们强调非单调激活函数（如sinc、Gaussian）作为频谱偏置的迭代高效替代方案具有实用价值。
原文摘要 · Abstract (English)
We generalize the connection between activation function and spline regression/smoothing and characterize how this choice may influence spectral bias within a 1D shallow network. We then demonstrate how gradient descent (GD) can be reinterpreted as a shrinkage operator that masks the singular values of a neural network's Jacobian. Viewed this way, GD implicitly selects the number of frequency components to retain, thereby controlling the spectral bias. An explicit relationship is proposed between the choice of GD hyperparameters (learning rate & number of iterations) and bandwidth (the number of active components). GD regularization is shown to be effective only with monotonic activation functions. Finally, we highlight the utility of non-monotonic activation functions (sinc, Gaussian) as iteration-efficient surrogates for spectral bias.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。