用傅里叶特征提升神经网络持续学习能力，避免遗忘。
Plastic Learning with Deep Fourier Features
- 引入深度傅里叶特征，每层融合正弦与余弦，平衡线性可训练性与非线性表达力。
- 在CIFAR10/100、tiny-ImageNet上，持续学习性能显著提升，抗标签噪声和类别增量干扰。
- 适用于各类持续学习场景，尤其适合需要长期稳定训练的模型部署。
深度神经网络在面对非平稳性时容易丧失持续学习能力，即‘塑性丢失’。本文揭示了保持塑性的基础原理，理论上证明线性函数近似及一类深层线性网络不会出现塑性丢失。随后提出深度傅里叶特征：每一层拼接正弦与余弦函数，实现线性带来的可训练性与非线性表达力之间的动态平衡。完全由深度傅里叶特征构成的网络具有高度可训练性，并在整个学习过程中保持稳定。实验证明，将ReLU替换为深度傅里叶特征可显著提升持续学习性能，涵盖标签噪声、类别增量学习、像素置换等多种场景，在CIFAR10、CIFAR100、tiny-ImageNet等主流持续学习数据集上均有效。
原文摘要 · Abstract (English)
Deep neural networks can struggle to learn continually in the face of non-stationarity. This phenomenon is known as loss of plasticity. In this paper, we identify underlying principles that lead to plastic algorithms. In particular, we provide theoretical results showing that linear function approximation, as well as a special case of deep linear networks, do not suffer from loss of plasticity. We then propose deep Fourier features, which are the concatenation of a sine and cosine in every layer, and we show that this combination provides a dynamic balance between the trainability obtained through linearity and the effectiveness obtained through the nonlinearity of neural networks. Deep networks composed entirely of deep Fourier features are highly trainable and sustain their trainability over the course of learning. Our empirical results show that continual learning performance can be drastically improved by replacing ReLU activations with deep Fourier features. These results hold for different continual learning scenarios (e.g., label noise, class incremental learning, pixel permutations) on all major supervised learning datasets used for continual learning research, such as CIFAR10, CIFAR100, and tiny-ImageNet.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。