arXiv:2608.12435cs.LG2026-08被引 1

用内容路由的锚点记忆扩展循环模型,让长文本记忆更准更快。

MARCH: Scaling Recurrent Memory with Content-Routed State Anchors

论文配图:MARCH: Scaling Recurrent Memory with Content-Routed State Anchors
图 1 · 摘自论文原文
  • 定期存档历史状态为锚点,用内容匹配快速查找
  • 在长文本任务上超越多个线性注意力模型,提升推理准确率
  • 适合需要高效长序列记忆的AI系统开发

Transformer依赖随上下文长度增长的逐标记记忆,但带来二次计算开销。循环模型虽能压缩历史信息保持效率,却因早期信息被覆盖而难以召回。本文提出MARCH架构:周期性缓存递归状态为状态锚点,并为每个锚点生成紧凑的内容条件键。由此构建可随上下文增长的记忆库,在历史分辨率与内存成本间实现可控权衡。每个词元通过锚点查询访问所有因果可用的锚点,输出为对历史锚点与当前状态的注意力聚合。预训练后,MARCH在常识推理、LongBench和上下文检索任务中持续优于多个线性注意力变体,证明内容路由的状态缓存显著增强循环长程记忆,同时保留原有计算路径。

原文摘要 · Abstract (English)

Transformers owe much of their strong long-context retrieval capability to a token-level memory that grows with context length. This flexibility, however, incurs a quadratic computation complexity during training and a key--value cache that grows linearly during autoregressive inference. Recurrent alternatives offer efficient decoding by compressing the entire history into a fixed-size state, but often underperform on recall-intensive tasks since earlier associations usually get overwritten by subsequent updates, and only the most recent contextual information is retained. In this paper, we introduce Memory-Anchor Routing across Context History (MARCH), a network architecture that effectively scales state-space models beyond a fixed-size dimension, while maintaining computational efficiency over long-sequences. MARCH periodically caches cumulative recurrent-state checkpoints as state anchors and associates each anchor with a compact, content-conditioned anchor key. This lets MARCH maintain a memory bank, which can grow as context length increases, providing a controllable trade-off between historical resolution and memory cost. At each token, MARCH produces an anchor query to attend all causally available state anchors, and the output is calculated as an attention-style aggregation over all historical anchors along the current state. We show that after standard pretraining, MARCH consistently outperforms multiple linear attention variants across commonsense reasoning, LongBench, and in-context retrieval. These results demonstrate that content-routed state caching substantially strengthens recurrent long-range memory while preserving its native computation path.

长序列建模状态空间模型记忆机制高效推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。