研究发现频率原则在球面上不总是成立，受初始条件影响
Is the Frequency Principle always valid?
- 用球谐展开分析神经网络学习动态，揭示频率偏好机制
- 固定或可训练权重下，低频先学但特定条件下会被打破
- 适合关注深度学习频率特性的研究人员参考
我们研究了浅层ReLU神经网络在三维空间单位球面$S^2$上极坐标$(τ,ϕ)$下的学习动态，分别考虑固定与可训练的神经元方向$\\(w_i\\)。对于固定权重，球谐展开显示内在的低频偏好，系数衰减为$O(\ell^{5/2}/2^\ell)$，通常导致频率原则（FP）中低频优先学习。然而，在特定初始条件或误差分布下，该原则可能被违背。对于可训练权重，谐波演化方程中新增旋转项，仍保持指数衰减，衰减速率为$O(\ell^{7/2}/2^\ell)$，同样支持低频优先学习。但同样存在例外情况。数值结果表明，可训练方向增加学习复杂性，既可维持低频优势，也可加速高频出现。这表明频率原则应视为一种趋势而非曲面域上的普适规律，为方向更新与谐波展开如何塑造频率依赖学习提供了新见解。
原文摘要 · Abstract (English)
We investigate the learning dynamics of shallow ReLU neural networks on the unit sphere \(S^2\subset\mathbb{R}^3\) in polar coordinates \((τ,ϕ)\), considering both fixed and trainable neuron directions \(\{w_i\}\). For fixed weights, spherical harmonic expansions reveal an intrinsic low-frequency preference with coefficients decaying as \(O(\ell^{5/2}/2^\ell)\), typically leading to the Frequency Principle (FP) of lower-frequency-first learning. However, this principle can be violated under specific initial conditions or error distributions. With trainable weights, an additional rotation term in the harmonic evolution equations preserves exponential decay with decay order \(O(\ell^{7/2}/2^\ell)\) factor, also leading to the FP of lower-frequency-first learning. But like fixed weights case, the principle can be violated under specific initial conditions or error distributions. Our numerical results demonstrate that trainable directions increase learning complexity and can either maintain a low-frequency advantage or enable faster high-frequency emergence. This analysis suggests the FP should be viewed as a tendency rather than a rule on curved domains like \(S^2\), providing insights into how direction updates and harmonic expansions shape frequency-dependent learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。