让神经网络只在有信息时更新,提升长序列建模效率与精度。
Efficient Sparse Selective-Update RNNs for Long-Range Sequence Modeling
- 通过神经元级开关只在关键时刻更新状态,避免冗余信息干扰。
- 在Long Range Arena等数据集上性能媲美甚至超越Transformer,且更高效。
- 适合需要长期记忆且计算资源有限的长序列任务,如语音、视频分析。
现实世界的序列信号(如音频或视频)中,关键信息常隐藏在长时间的静音或噪声中。虽然循环神经网络(RNN)旨在高效处理此类数据,但其固定的更新机制——每步必更新——导致内存衰减:即使输入无变化,模型仍持续覆盖记忆,使学习信号难以回溯到遥远过去。本文提出选择性更新RNN(suRNN),一种非线性架构,可学习在输入冗余时保留记忆。通过神经元级别的二值开关,仅在有信息事件时开启更新,使递归更新与原始序列长度解耦。该机制在低信息时段保持精确不变的记忆,为梯度提供了跨时间的直接通路。在Long Range Arena、WikiText及其他合成基准上的实验表明,suRNNs在准确率上达到或超过复杂模型(如Transformer),同时对长期存储更具效率。通过让每个神经元自主学习更新周期,本方法解决了序列长度与实际信息量之间的不匹配问题。通过提供管理时间信息密度的原理性方案,本工作为在高效循环框架内实现类Transformer性能开辟了新路径。
原文摘要 · Abstract (English)
Real-world sequential signals, such as audio or video, contain critical information that is often embedded within long periods of silence or noise. While recurrent neural networks (RNNs) are designed to process such data efficiently, they often suffer from ``memory decay'' due to a rigid update schedule: they typically update their internal state at every time step, even when the input is static. This constant activity forces the model to overwrite its own memory and makes it hard for the learning signal to reach back to distant past events. Here we show that we can overcome this limitation using Selective-Update RNNs (suRNNs), a non-linear architecture that learns to preserve its memory when the input is redundant. By using a neuron-level binary switch that only opens for informative events, suRNNs decouple the recurrent updates from the raw sequence length. This mechanism allows the model to maintain an exact, unchanged memory of the past during low-information intervals, creating a direct path for gradients to flow across time. Our experiments on the Long Range Arena, WikiText, and other synthetic benchmarks show that suRNNs match or exceed the accuracy of much more complex models such as Transformers, while remaining significantly more efficient for long-term storage. By allowing each neuron to learn its own update timescale, our approach resolves the mismatch between how long a sequence is and how much information it actually contains. By providing a principled approach to managing temporal information density, this work establishes a new direction for achieving Transformer-level performance within the highly efficient framework of recurrent modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。