arXiv:2410.16105cs.LGcs.AI2024-10NeurIPS被引 22

用多级浅层网络组合,提升深度网络对高频特征的捕捉能力。

Addressing Spectral Bias of Deep Neural Networks by Multi-Grade Deep Learning

  • 分阶段训练浅层网络,逐级学习低频信息并组合成高阶表示。
  • 在合成数据、流形数据和图像上均显著提升高频特征建模效果。
  • 适合需要精细细节建模的任务,如图像重建与纹理生成。

深度神经网络(DNN)存在谱偏差问题,即倾向于优先学习函数的低频成分,难以捕捉高频特征。本文提出通过多级深度学习(MGDL)模型解决该问题:利用多个浅层神经网络(SNN)分别学习数据中的低频信息,并将它们的组合用于逼近高频率函数。MGDL采用渐进式训练策略,每一级仅基于前一级残差进行学习,并将此前所有层级的SNN作为特征输入。实验在包含高频特征的合成数据、流形数据、彩色图像及MNIST数据集上进行,结果表明,每级网络都能有效捕获低频成分,其组合可精准表征高频结构。该方法显著缓解了传统DNN的谱偏差,为提升深度学习模型在高频信息任务中的表现提供了新思路。

原文摘要 · Abstract (English)

Deep neural networks (DNNs) suffer from the spectral bias, wherein DNNs typically exhibit a tendency to prioritize the learning of lower-frequency components of a function, struggling to capture its high-frequency features. This paper is to address this issue. Notice that a function having only low frequency components may be well-represented by a shallow neural network (SNN), a network having only a few layers. By observing that composition of low frequency functions can effectively approximate a high-frequency function, we propose to learn a function containing high-frequency components by composing several SNNs, each of which learns certain low-frequency information from the given data. We implement the proposed idea by exploiting the multi-grade deep learning (MGDL) model, a recently introduced model that trains a DNN incrementally, grade by grade, a current grade learning from the residue of the previous grade only an SNN composed with the SNNs trained in the preceding grades as features. We apply MGDL to synthetic, manifold, colored images, and MNIST datasets, all characterized by presence of high-frequency features. Our study reveals that MGDL excels at representing functions containing high-frequency information. Specifically, the neural networks learned in each grade adeptly capture some low-frequency information, allowing their compositions with SNNs learned in the previous grades effectively representing the high-frequency features. Our experimental results underscore the efficacy of MGDL in addressing the spectral bias inherent in DNNs. By leveraging MGDL, we offer insights into overcoming spectral bias limitation of DNNs, thereby enhancing the performance and applicability of deep learning models in tasks requiring the representation of high-frequency information. This study confirms that the proposed method offers a promising solution to address the spectral bias of DNNs.

谱偏差多级学习高频建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。