调优RNN训练中的预热阶段，可让预测误差降低60%以上
Tuning the burn-in phase in training recurrent neural networks improves their performance
- 通过截断反向传播优化子序列，减少计算开销
- 理论证明子序列训练存在性能损失，与预热阶段密切相关
- 实验证明合理设置预热期能显著提升时间序列预测效果
使用标准的随时间反向传播（BPTT）训练循环神经网络（RNN）在长输入序列下尤为困难。为降低计算和内存开销，一种实用方法是将训练数据分段进行多次截断BPTT。本文研究了此类截断学习方法在时间序列任务中的应用。具体地，我们建立了在子序列上优化而非全序列优化时的准确率与性能损失的理论边界。结果表明，RNN的烧入阶段（burn-in phase）是影响性能保证的重要调节参数。我们在系统辨识和时间序列预测的标准基准上验证了理论结果。所有实验均显示烧入阶段对训练过程有显著影响，合理调优可使训练与测试数据上的预测误差降低超过60%。
原文摘要 · Abstract (English)
Training recurrent neural networks (RNNs) with standard backpropagation through time (BPTT) can be challenging, especially in the presence of long input sequences. A practical alternative to reduce computational and memory overhead is to perform BPTT repeatedly over shorter segments of the training data set, corresponding to truncated BPTT. In this paper, we examine the training of RNNs when using such a truncated learning approach for time series tasks. Specifically, we establish theoretical bounds on the accuracy and performance loss when optimizing over subsequences instead of the full data sequence. This reveals that the burn-in phase of the RNN is an important tuning knob in its training, with significant impact on the performance guarantees. We validate our theoretical results through experiments on standard benchmarks from the fields of system identification and time series forecasting. In all experiments, we observe a strong influence of the burn-in phase on the training process, and proper tuning can lead to a reduction of the prediction error on the training and test data of more than 60% in some cases.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。