用记忆标签训练RNN，突破传统序列依赖限制。
Pretraining Recurrent Networks without Recurrence
- 将RNN训练转为一步记忆预测的监督学习
- 实现稳定梯度传播，长程依赖建模能力更强
- 适合需要并行训练和长期记忆的序列模型
训练循环神经网络（RNN）需在长序列中分配梯度信用，标准反向传播通过时间（BPTT）方法因串行计算、梯度消失或爆炸问题而效率低下。本文提出有监督记忆训练（SMT），通过训练基于Transformer的编码器以预测未来状态，生成每一步的记忆标签（m_t, x_{t+1}）→ m_{t+1}，从而完全绕过递归梯度传播。SMT分离了记忆内容与更新机制，使RNN训练可并行进行，任意两词间梯度路径保持恒定复杂度O(1)，无需展开网络。实验表明，在语言建模和像素序列建模任务上，SMT优于BPTT，显著提升非线性RNN对长程依赖的捕捉能力，为构建具有时间抽象能力的可扩展模型提供新路径。
原文摘要 · Abstract (English)
Training recurrent neural networks (RNNs) requires assigning credit across long sequences of computations. Standard backpropagation through time (BPTT) addresses this problem poorly: it is sequential in time, limiting parallelism, and suffers from vanishing or exploding gradients, making long-range associations difficult to learn. We propose Supervised Memory Training (SMT), a method for training nonlinear RNNs that sidesteps recurrent credit propagation entirely by reducing RNN training to supervised learning on one-step memory transition labels $(m_t, x_{t+1}) \rightarrow m_{t+1}$. SMT acquires these memory labels by training a Transformer-based encoder on a predictive state objective--retaining only information from the past necessary to predict the future. By decoupling what to remember from how to update memory, SMT enables time-parallel RNN training with a stable $O(1)$ length gradient path between any two tokens--without ever unrolling the RNN. We find that SMT outperforms BPTT when pretraining various RNN architectures on tasks like language modeling and pixel sequence modeling. SMT enables nonlinear RNNs to better capture long-range dependencies and train in parallel, potentially unlocking the scaling of models that build temporal abstractions of past experience.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。