用固定反馈机制替代BPTT，让RNN训练更快更高效
Fast Training of Recurrent Neural Networks with Stationary State Feedbacks
- 用状态空间模型设计结构化反馈矩阵，直接传递未来梯度
- 在语言建模任务上达到与原模型相当的困惑度，但训练成本大幅降低
- 适合追求高效训练的RNN应用，尤其看重推理速度的场景
循环神经网络（RNN）在参数量相近的情况下，表现强于Transformer且推理更快。然而，基于时间反向传播（BPTT）的递归梯度计算仍是主要计算瓶颈。本文提出一种新方法，用固定梯度反馈机制替代BPTT，基于时间平稳性假设，近似精确梯度传播。该方法借鉴状态空间模型（SSM）原理，定义结构化反馈矩阵，直接从未来时间步传播梯度，无需递归反向传播，显著降低训练开销，同时保持捕捉长期依赖的能力。在语言建模基准测试中，模型取得具有竞争力的困惑度分数，同时大幅减少训练成本。结果表明，类似SSM的反馈设计能充分释放RNN在实际应用中的效率优势。
原文摘要 · Abstract (English)
Recurrent neural networks (RNNs) have recently demonstrated strong performance and faster inference than Transformers at comparable parameter budgets. However, the recursive gradient computation with the backpropagation through time (or BPTT) algorithm remains the major computational bottleneck. In this work, we propose a novel method that replaces BPTT with a fixed gradient feedback mechanism, yielding an efficient approximation of the exact gradient propagation based on the assumption of time stationarity. Our approach leverages state-space model (SSM) principles to define a structured feedback matrix that directly propagates gradients from future time steps. This formulation bypasses the need for recursive gradient backpropagation, significantly reducing training overhead while preserving the network's ability to capture long-term dependencies. The experiments on language modeling benchmarks exhibit competitive perplexity scores, while significantly reducing the training costs. These promising results suggest that designing a feedback method like an SSM can fully exploit the efficiency advantages of RNNs for many practical applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。