arXiv:2605.06946cs.LGcs.AI2026-05

让注意力记忆衰减随内容自适应,提升长序列建模能力。

Adaptive Memory Decay for Log-Linear Attention

论文配图:Adaptive Memory Decay for Log-Linear Attention
图 1 · 摘自论文原文
  • 用轻量MLP根据输入动态学习记忆衰减参数
  • 在长序列任务中显著提升记忆保持效果
  • 适合需要长程依赖建模的NLP任务

序列模型面临内存容量与计算效率之间的根本权衡。Transformer以二次复杂度实现丰富的上下文建模,而线性注意力和状态空间模型通过将上下文压缩到固定大小的隐藏状态,在线性时间内运行,但本质上限制了记忆回溯能力。对数线性注意力通过分层芬威克树组织记忆,在日志线性计算成本下使隐藏状态随序列长度对数增长。然而,其记忆衰减参数λ为固定值且与输入无关,导致各层级权重分配均匀,缺乏内容敏感性,引入不必要的刚性。本文提出通过轻量级两层MLP从输入直接学习λ,实现逐标记、逐层级的自适应衰减,使其更贴合内容而非位置。软加激活函数使各层级可独立缩放,避免了softmax带来的层级间竞争。该改进完全保持对数线性复杂度,参数开销可忽略。在关联回忆、选择性复制和语言建模任务上的评估表明,输入相关衰减始终优于基线,在长程记忆场景下表现最优,此时基线λ会退化甚至崩溃。

原文摘要 · Abstract (English)

Sequence models face a fundamental tradeoff between memory capacity and computational efficiency. Transformers achieve expressive context modeling at quadratic cost, while linear attention and state-space models run in linear time by compressing context into a fixed-size hidden state, inherently limiting recall. Log-linear attention navigates this tradeoff by organizing memory across a Fenwick tree hierarchy, growing its hidden state logarithmically with sequence length at log-linear compute cost. However, its memory decay parameter λ is fixed and independent of the input, assigning uniform weights across all hierarchy levels regardless of the content, which introduces unnecessary rigidity. We propose learning λ directly from the input via a lightweight two-layer MLP, producing per-token, per-level decay that adapts to content rather than position. A softplus activation lets each Fenwick tree level scale independently, avoiding the inter-level competition that softmax introduces. This modification preserves log-linear complexity exactly and adds negligible parameter overhead. We evaluate on associative recall, selective copying, and language modeling, finding that input-dependent decay consistently outperforms the baseline, with the largest gains in long-range memory settings where baseline λ degrades or collapses entirely.

注意力机制长序列建模自适应衰减线性注意力

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。