简化GRU结构,提升持续学习效率与部署可行性
Minion Gated Recurrent Unit for Continual Learning
- 用缩放系数替代传统门控,降低计算与内存开销
- 训练速度提升2.90倍,参数量减少2.88倍,性能相当
- 适合资源受限设备,尤其在持续学习中表现稳定
序列数据处理对持续学习的需求日益增长,导致训练方法和循环神经网络(RNN)架构日趋复杂,加剧了其在低内存、低算力设备上的应用差距。为此,我们研究了简化RNN架构(特别是门控循环单元,GRU)的有效性及其对单任务与多任务序列学习的影响。提出一种新型GRU变体——极小循环单元(MiRU),用缩放系数替代传统门控机制,以调节隐藏状态和历史上下文的动态更新,显著降低计算成本与内存占用。尽管结构简化,MiRU在序列图像分类与自然语言处理基准测试中仍保持与标准GRU相当的性能,训练速度提升2.90倍,参数量减少2.88倍。通过基于回放策略与全局抑制的持续学习实验发现,仅使用回放时MiRU在多任务学习中仍能保持稳定表现,而标准GRU及其变体则出现性能下降。这些特性使MiRU成为边缘设备应用的有力候选。
原文摘要 · Abstract (English)
The increasing demand for continual learning in sequential data processing has led to progressively complex training methodologies and larger recurrent network architectures. Consequently, this has widened the knowledge gap between continual learning with recurrent neural networks (RNNs) and their ability to operate on devices with limited memory and compute. To address this challenge, we investigate the effectiveness of simplifying RNN architectures, particularly gated recurrent unit (GRU), and its impact on both single-task and multitask sequential learning. We propose a new variant of GRU, namely the minion recurrent unit (MiRU). MiRU replaces conventional gating mechanisms with scaling coefficients to regulate dynamic updates of hidden states and historical context, reducing computational costs and memory requirements. Despite its simplified architecture, MiRU maintains performance comparable to the standard GRU while achieving 2.90x faster training and reducing parameter usage by 2.88x, as demonstrated through evaluations on sequential image classification and natural language processing benchmarks. The impact of model simplification on its learning capacity is also investigated by performing continual learning tasks with a rehearsal-based strategy and global inhibition. We find that MiRU demonstrates stable performance in multitask learning even when using only rehearsal, unlike the standard GRU and its variants. These features position MiRU as a promising candidate for edge-device applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。