揭示深度网络隐含的低频偏好,解释正则化为何提升泛化能力
Frequency Regularization: Unveiling the Spectral Inductive Bias of Deep Neural Networks
- 通过频谱抑制比量化正则化对高频成分的压制强度
- L2正则使高频能量降低超3倍,但加剧对高斯噪声敏感度
- 首次发现低频偏好带来抗模糊优势,适合图像退化场景
L2正则与Dropout是训练深度神经网络的基础技术,但其在特征频率选择上的物理机制尚不清晰。本文研究现代卷积神经网络的频谱偏差,提出可视化诊断框架追踪权重频谱演化,并引入频谱抑制比(SSR)量化不同正则化器的“低通滤波”强度。针对小卷积核(如3×3)的混叠问题,采用离散径向剖分方法,在ResNet-18与CIFAR-10上实验证明:相比无正则基线,L2正则使高频能量积累抑制超过3倍。进一步发现精度-鲁棒性权衡:虽因过度专注低频而更易受宽带高斯噪声影响,但在低分辨率等高频信息丢失场景中,其性能优于基线超过6%。本工作从信号处理视角揭示正则化强制低频结构的谱归纳偏置,支持其泛化作用。
原文摘要 · Abstract (English)
Regularization techniques such as L2 regularization (Weight Decay) and Dropout are fundamental to training deep neural networks, yet their underlying physical mechanisms regarding feature frequency selection remain poorly understood. In this work, we investigate the Spectral Bias of modern Convolutional Neural Networks (CNNs). We introduce a Visual Diagnostic Framework to track the dynamic evolution of weight frequencies during training and propose a novel metric, the Spectral Suppression Ratio (SSR), to quantify the "low-pass filtering" intensity of different regularizers. By addressing the aliasing issue in small kernels (e.g., 3x3) through discrete radial profiling, our empirical results on ResNet-18 and CIFAR-10 demonstrate that L2 regularization suppresses high-frequency energy accumulation by over 3x compared to unregularized baselines. Furthermore, we reveal a critical Accuracy-Robustness Trade-off: while L2 models are sensitive to broadband Gaussian noise due to over-specialization in low frequencies, they exhibit superior robustness against high-frequency information loss (e.g., low resolution), outperforming baselines by >6% in blurred scenarios. This work provides a signal-processing perspective on generalization, confirming that regularization enforces a strong spectral inductive bias towards low-frequency structures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。