arXiv:2504.13531cs.LG2025-04

提出局部表示对齐RNN,改善梯度不稳定问题。

Can Local Representation Alignment RNNs Solve Temporal Tasks?

  • 用局部更新替代BPTT,通过表示对齐减少梯度异常。
  • 引入梯度正则化后,模型在3个时序任务上显著提升性能。
  • 适用于需稳定训练的实时时序处理场景,如机器人控制。

循环神经网络(RNN)常用于实时处理、流数据及样本有限的情况。传统反向传播通过时间(BPTT)易导致梯度爆炸或消失,且生物合理性存疑。本文提出基于目标传播的RNN训练方法,采用局部表示对齐(LRA),通过局部更新缓解梯度不稳定性。实验分析了归一化与不同局部误差函数的影响,发现尽管网络被分解为子图,梯度仍会消失;且原方案中提出的梯度裁剪对性能影响甚微。为此,我们引入沿更新方向的梯度正则化,有效促进梯度流动并提升收敛性。在三个基准时序任务——时序顺序、3位时序顺序和随机置换上,正则化LRA RNN显著优于未正则化版本。

原文摘要 · Abstract (English)

Recurrent Neural Networks (RNNs) are commonly used for real-time processing, streaming data, and cases where the amount of training samples is limited. Backpropagation Through Time (BPTT) is the predominant algorithm for training RNNs; however, it is frequently criticized for being prone to exploding and vanishing gradients and being biologically implausible. In this paper, we present and evaluate a target propagation-based method for RNNs, which uses local updates and seeks to reduce the said instabilities. Having stable RNN models increases their practical use in a wide range of fields such as natural language processing, time-series forecasting, anomaly detection, control systems, and robotics. The proposed solution uses local representation alignment (LRA). We thoroughly analyze the performance of this method, experiment with normalization and different local error functions, and invalidate certain assumptions about the behavior of this type of learning. Namely, we demonstrate that despite the decomposition of the network into sub-graphs, the model still suffers from vanishing gradients. We also show that gradient clipping as proposed in LRA has little to no effect on network performance. This results in an LRA RNN model that is very difficult to train due to vanishing gradients. We address this by introducing gradient regularization in the direction of the update and demonstrate that this modification promotes gradient flow and meaningfully impacts convergence. We compare and discuss the performance of the algorithm, and we show that the regularized LRA RNN considerably outperforms the unregularized version on three landmark tasks: temporal order, 3-bit temporal order, and random permutation.

RNN梯度稳定时序建模局部学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。