LSTM训练中出现多次性能波动，源于有序到混沌的相变过程。
Multiple Descents in Deep Learning as a Sequence of Order-Chaos Transitions in LSTM Networks
- 通过渐近稳定性分析发现性能波动与模型有序-混沌相变相关。
- 最优训练步长始终位于有序与混沌的临界过渡点上。
- 首次从有序到混沌的过渡阶段最适合作为最佳训练点。
我们在真实任务上训练长短期记忆（LSTM）网络时，观察到一种新型的‘多峰下降’现象：模型过拟合后，性能在测试损失上反复经历多次上下波动。通过渐近稳定性分析，我们发现这些性能周期性波动与模型在有序与混沌之间的相变过程密切相关。局部最优训练步骤始终处于两个相态的临界过渡点。更重要的是，模型最佳表现通常出现在首次从有序向混沌转变的阶段，此时‘混沌边缘’的宽度最大，有利于权重配置的充分探索，从而实现最优学习。该现象揭示了深度学习优化过程中的动态机制。
原文摘要 · Abstract (English)
We observe a novel `multiple-descent' phenomenon during the learning process of a recurrent neural network called long-short-term memory (LSTM) networks during its training on real-world task, in which the performance goes through long cycles of up and down trends multiple times after the model is overtrained. By carrying out asymptotic stability analysis of the models, we found that the cycles in performance -- indicated by loss function in test data -- are closely associated with the phase transition process between order and chaos of the model, and the local optimal training step are consistently at the critical transition point between the two phases. More importantly, the most optimal point of the model usually occurs at the first transition from order to chaos, where the `width' of the `edge of chaos' is often the widest, allowing the best exploration of weight configurations for learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。