用睡眠式回放机制,让模型高效记住长期非平稳时序模式。
SHARP: Sleep-based Hierarchical Accelerated Replay for Long Range Non-Stationary Temporal Pattern Recognition
- 分层记忆+离线加速回放,避免长时间反向传播
- 在text8和PG-19上保持旧数据预测能力并学习新流
- 适合处理持续输入的长时序任务,如在线文本生成
学习长程非平稳时序模式仍是现代序列模型的核心挑战,尤其在严格流式场景中。数据按顺序到达且需单次遍历处理,无法同时回顾历史。标准架构如循环神经网络和Transformer受限于截断的反向传播时间范围或显式的输入窗口长度。为此,我们提出SHARP(基于睡眠的分层加速回放)框架,将时序学习分解为两个互补组件:记忆模块累积结构化历史输入,模式识别模块在此记忆上运行。该分离设计通过消除跨多步的反向传播需求,实现资源与计算效率的提升。受啮齿类动物慢波睡眠期间加速回放现象启发,SHARP引入离线(睡眠)阶段,以加速形式重播时序结构化的记忆痕迹,并整合至高层记忆表示,增强长程上下文保留。通过受控模拟与消融研究,我们刻画了该框架的关键特性。在text8和PG-19等基准数据集上,SHARP在持续学习当前流的同时,仍能保持对已见数据的下一个词预测性能,并泛化到未来未见数据,显著优于循环基线。这些提升得益于其分层结构,仅以线性时间复杂度实现指数级增长的有效时序上下文。
原文摘要 · Abstract (English)
Learning long-range non-stationary temporal patterns remains a core challenge for modern sequence models, particularly in strict streaming settings. In these settings, data arrive sequentially and must be processed in a single pass without simultaneously revisiting past observations. Standard architectures, including recurrent neural networks and transformers, are constrained by either truncated backpropagation through time horizon or explicit input window length for long range credit assignment. To address these limitations, we propose SHARP (Sleep-based Hierarchical Accelerated Replay), a framework that decomposes temporal learning into two complementary components: a memory module that accumulates a structured history of past inputs, and a pattern-recognition module that operates over this memory. This separation enables resource- and compute-efficient adaptation to non-stationary dynamics by eliminating the need for backpropagation through time across many steps for long-range credit assignment. Inspired by the accelerated replay observed in rodents during slow-wave sleep, SHARP incorporates offline (sleep) phases in which temporally structured memory traces are replayed in an accelerated form and integrated into higher-level memory representations, improving long-range context retention. Through controlled simulations and ablation studies, we characterize the key properties of the proposed framework. In benchmark datasets such as text8 and PG-19, we demonstrate that SHARP improves over recurrent baselines by retaining next-token predictive performance on previously seen data while continuing to learn from the current stream and generalizing to future unseen data. These gains are enabled by its hierarchical structure, which yields an exponentially increasing effective temporal context with only linear-time computational cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。