通过记忆神经元状态，稳定长期动态神经网络训练
The Lifecycle Principle: Stabilizing Dynamic Neural Networks with State Memory
- 让神经元休眠后恢复时保留最后有效参数，避免随机初始化
- 在图像分类任务中提升泛化能力，显著增强模型鲁棒性
- 适合需要高稳定性与强泛化能力的深度学习场景
我研究了一种更强的正则化形式：将神经元长时间关闭，区别于Dropout等临时扰动方法。然而，这种长期动态性带来了严重挑战——神经元重启时若用随机权重初始化，会导致训练极度不稳定。为此，我提出生命周期（LC）原则，核心创新是引入状态记忆机制：神经元复活时不重置参数，而是恢复到其上一次有效状态。该机制保留了已学知识，避免优化过程中的剧烈冲击。理论分析表明，LC原则使损失曲面更平滑，引导优化趋向平坦极小值，从而提升泛化性能。在图像分类基准测试中，该方法显著改善了模型的泛化与鲁棒性。关键消融实验进一步验证，状态记忆是实现这些优势的关键。
原文摘要 · Abstract (English)
I investigate a stronger form of regularization by deactivating neurons for extended periods, a departure from the temporary changes of methods like Dropout. However, this long-term dynamism introduces a critical challenge: severe training instability when neurons are revived with random weights. To solve this, I propose the Lifecycle (LC) principle, a regularization mechanism centered on a key innovation: state memory. Instead of re-initializing a revived neuron, my method restores its parameters to their last known effective state. This process preserves learned knowledge and avoids destructive optimization shocks. My theoretical analysis reveals that the LC principle smooths the loss landscape, guiding optimization towards flatter minima associated with better generalization. Experiments on image classification benchmarks demonstrate that my method improves generalization and robustness. Crucially, ablation studies confirm that state memory is essential for achieving these gains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。