从傅里叶视角揭示神经网络如何先学幅度后学相位，解释图像学习效率的机制。
A Fourier perspective on the learning dynamics of neural networks: from sample complexities to mechanistic insights
- 用傅里叶分析自然图像的平移不变性与幂律谱，建模幅度与相位的可学习性差异。
- 相位信息单独分类在高维下极难学习，需超过 $N^3 \log^2 N$ 步，远超幅度学习。
- 幂律谱能加速相位学习，即使不提升分类性能，适合研究深层网络机制的人参考。
基于梯度的方法训练的神经网络表现出强简化偏倚：优先学习数据中更简单的统计特征，再处理复杂特征。以往分析多限于（准）各向同性输入。本文从傅里叶视角出发，引入自然图像的两个关键特性——近似平移不变性和幂律谱。实验表明，图像分类任务中简单网络先依赖像素对相关性的幅度信息，后才利用编码边缘和高阶相关性的相位信息。为此，我们提出一种可精确控制幅度与相位的平移不变合成数据模型。理论证明：对于各向同性高维输入，仅靠相位进行分类是真正困难的任务——在线随机梯度下降（SGD）在 $n \ll N^3$ 步内无法区分结构化输入与噪声，至少需要 $n \gg N^3 \log^2 N$ 步。相反，实验与理论均显示，幂律谱能显著加速相位学习，即便其本身不提升分类性能。两层网络在纹理数据、深度卷积网络在 ImageNet 与 CIFAR100 上的模拟验证了幅度与相位间的非平凡交互，为深层网络高效学习自然图像分布提供了机制洞察。
原文摘要 · Abstract (English)
Neural networks trained with gradient-based methods exhibit a strong simplicity bias: they learn simpler statistical features of their data before moving to more complex features. Previous analyses of this phenomenon have largely focused on settings with (quasi-)isotropic inputs. In this work, we study the simplicity bias from a Fourier perspective, which allows us to include two key features of natural images in the analysis: approximate translation-invariance and power-law spectra. We first show experimentally that simple neural networks trained on image classification tasks first rely on amplitude information -- related to pair-wise correlations between pixels -- before exploiting phase information, which encodes edges and higher-order correlations. In view of this, we introduce a synthetic data model for translation-invariant inputs that allows precise control over amplitudes and phases while remaining tractable. We rigorously establish that for isotropic and high-dimensional inputs, classification based on phase information alone is a genuinely hard task: online stochastic gradient descent (SGD) cannot distinguish the structured inputs from noise within $n \ll N^3$ steps, but needs at least $n \gg N^3 \log^2{N}$ steps. In contrast, we show both experimentally and theoretically that power-law spectra can dramatically accelerate the speed of learning phase information, even if the spectra do not help with classification. Simulations with two-layer networks trained on textures and with deep convolutional networks on ImageNet and CIFAR100 confirm this non-trivial interaction between amplitudes and phases, providing mechanistic insights into how deep neural networks can learn natural image distributions efficiently.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。