提出DASH方法，解决模型冷启动时遗忘旧知识的问题。
DASH: Warm-Starting Neural Network Training in Stationary Settings without Loss of Plasticity
- 通过识别噪声记忆是导致学习能力下降的主因，设计选择性遗忘机制。
- 在视觉任务上实现测试准确率提升和训练效率优化。
- 适合持续学习场景下需要保持学习能力的神经网络应用。
通过使用先前学习的权重初始化神经网络来热启动训练具有吸引力，因为实际应用中的神经网络通常面临持续的新数据输入。然而，这常常导致学习能力丧失，即网络失去学习新信息的能力，从而导致泛化性能劣于从零开始训练。这种情况即使在数据分布静止时也会发生，其内在机制尚不明确。我们构建了一个模拟真实神经网络训练的框架，发现噪声记忆是热启动时学习能力丧失的主要原因。受此启发，我们提出了方向感知收缩（DASH）方法，旨在通过选择性遗忘已记忆的噪声，同时保留已学习的特征，以缓解学习能力丧失问题。我们在视觉任务上验证了该方法的有效性，结果表明其在测试准确率和训练效率方面均有显著提升。
原文摘要 · Abstract (English)
Warm-starting neural network training by initializing networks with previously learned weights is appealing, as practical neural networks are often deployed under a continuous influx of new data. However, it often leads to loss of plasticity, where the network loses its ability to learn new information, resulting in worse generalization than training from scratch. This occurs even under stationary data distributions, and its underlying mechanism is poorly understood. We develop a framework emulating real-world neural network training and identify noise memorization as the primary cause of plasticity loss when warm-starting on stationary data. Motivated by this, we propose Direction-Aware SHrinking (DASH), a method aiming to mitigate plasticity loss by selectively forgetting memorized noise while preserving learned features. We validate our approach on vision tasks, demonstrating improvements in test accuracy and training efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。