arXiv:2608.16844cs.LGcs.AI2026-08

通过渐进激活记忆,提升长文本建模的效率与准确性。

Proteus: Incremental Memory Activation for Long-Context Sequence Modeling

论文配图:Proteus: Incremental Memory Activation for Long-Context Sequence Modeling
图 1 · 摘自论文原文
  • 记忆容量随上下文增长逐步释放,早期内容压缩更充分。
  • 在长文本任务中,准确率提升显著,长度越长效果越明显。
  • 可无缝集成到多种模型,无需额外计算开销。

基于注意力的序列模型在处理长上下文时存在二次复杂度问题,促使研究者探索将上下文压缩为紧凑状态的记忆模型。然而,现有模型通常在整个序列中使用静态记忆,导致早期令牌因无压缩压力而占据过多自由度,污染记忆状态,使后续内容存储干扰加剧。本文提出增量记忆激活新范式:随着上下文增长,逐步扩展记忆的有效容量。早期施加瓶颈迫使模型更高效地压缩历史,后期释放新容量则减少干扰,提升后期内容保留能力。我们构建了Proteus机制,可无缝嵌入多种神经记忆架构,且无需额外开销。在SWLA、Comba、Titans和Hope-Attention等先进模型上应用,均在标准语言建模、推理、长文本检索与理解任务中取得一致提升,增益随上下文长度增加而扩大。结果表明,静态记忆非最优,调度有效容量是一种简单且普适的序列建模工具。

原文摘要 · Abstract (English)

The quadratic cost of attention-based sequence models for long contexts has motivated a growing line of research on memory-based models that can compress context into a compact state. However, most existing memory models expose a static memory throughout the entire sequence. Because early tokens face no compression pressure, they occupy too many degrees of freedom and "pollute" the memory state, leaving little capacity for later context and increasing interference between what is stored and what arrives next. We study a new paradigm of incremental memory activation, where the effective capacity of memory is progressively expanded as the context grows. Imposing an early bottleneck forces the model to compress history more effectively, while unlocking fresh capacity over time reduces interference and improves retention of later context. We instantiate this paradigm in Proteus, a straightforward mechanism that can be incorporated into a broad class of neural memory architectures at no additional cost. We apply Proteus to state-of-the-art models, including SWLA, Comba, Titans, and Hope-Attention, and observe consistent improvements on standard language modeling and reasoning, as well as on long-context retrieval and understanding, with gains that grow at longer context lengths. Overall, our results show that static memory is suboptimal and that scheduling effective capacity is a simple and broadly applicable tool for sequence modeling.

长文本建模记忆机制序列模型增量学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。