简化LSTM和GRU，实现训练并行化与高性能。
Were RNNs All We Needed?
- 精简LSTM/GRU结构，减少参数量
- 训练时完全并行，突破传统RNN瓶颈
- 性能媲美Transformer，适合长序列场景
2017年Transformer的出现重塑了深度学习格局。尽管其在序列建模中表现卓越，但受限于序列长度扩展性，促使研究者重新关注可并行训练的新一代循环模型。本文从历史视角回顾循环神经网络（RNN）的发展，聚焦曾主导领域二十年的LSTM（1997）与GRU（2014）。通过简化模型结构，我们提出minLSTM与minGRU，二者不仅参数量少于传统版本，且在训练过程中可完全并行化，同时在多项任务上表现出令人意外的竞争力，性能可与近期Transformer模型相媲美。
原文摘要 · Abstract (English)
The introduction of Transformers in 2017 reshaped the landscape of deep learning. Originally proposed for sequence modelling, Transformers have since achieved widespread success across various domains. However, the scalability limitations of Transformers - particularly with respect to sequence length - have sparked renewed interest in novel recurrent models that are parallelizable during training, offer comparable performance, and scale more effectively. In this work, we revisit sequence modelling from a historical perspective, focusing on Recurrent Neural Networks (RNNs), which dominated the field for two decades before the rise of Transformers. Specifically, we examine LSTMs (1997) and GRUs (2014). We demonstrate that by simplifying these models, we can derive minimal versions (minLSTMs and minGRUs) that (1) use fewer parameters than their traditional counterparts, (2) are fully parallelizable during training, and (3) achieve surprisingly competitive performance on a range of tasks, rivalling recent models including Transformers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。