arXiv:2606.10650cs.CLcs.AI2026-06中稿 · ICML被引 2

动态调整注意力状态边界,提升长文本建模精度。

Dynamic Linear Attention

论文配图:Dynamic Linear Attention
图 1 · 摘自论文原文
  • 根据词元信息变化动态决定状态合并时机,避免关键信息丢失。
  • 固定大小缓存机制控制内存增长,仅损失少量信息。
  • 适用于需要处理长序列的LLM场景,如文档理解与生成。

大语言模型在长上下文中的扩展性受限于标准注意力机制的二次复杂度,促使采用线性注意力以降低计算开销。为提升长序列下的表征能力,现有方法将记忆组织为多状态形式。然而,现有多状态线性注意力方法依赖固定的状态合并策略,无法适应动态变化的词元重要性,导致关键词元被不可逆地掩盖,并在长序列中引发严重误差累积。为此,本文提出DLA(动态线性注意力)框架,引入两项创新:(i) 信息感知的动态状态合并,基于词元级信息变化自适应确定状态边界,在语义转换处保留高分辨率表示,同时对稳定区域进行高效摘要;(ii) 有容量限制的记忆建模,通过选择性合并相邻低信息量状态,维持固定大小且按时间顺序排列的状态缓存,有效控制内存增长并最小化信息损失。我们在两种不同的线性注意力模型上预训练DLA,并在三大类共16个数据集上评估。实验结果表明,DLA在性能上优于当前最优方法。

原文摘要 · Abstract (English)

The scalability of Large Language Models (LLMs) to long contexts is fundamentally constrained by the quadratic complexity of standard attention, motivating the adoption of linear attention mechanisms with sub-quadratic cost. To improve representation capacity under long contexts, recent approaches organize memory in a multi-state manner. However, existing multi-state linear attention methods rely on fixed state merging policies that cannot adapt to dynamically varying token importance, irreversibly obscuring critical tokens and causing severe error accumulation over long sequences. To address this limitation, we propose DLA, a dynamic memory modeling framework for multi-state linear attention. DLA introduces (i) Information-Aware Dynamic State Merging, which adaptively determines state boundaries based on token-level information variation, preserving high-resolution representations around semantic transitions while aggressively summarizing stable regions, and (ii) Capacity-Bounded Memory Modeling, which maintains a fixed-size, chronologically ordered state cache by selectively merging adjacent low-information states to control memory growth with minimal information loss. We pre-train DLA on two different linear attention models and evaluate on 16 datasets across three categories. Experimental results demonstrate the superiority of DLA over state-of-the-art.

注意力机制长序列LLM优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。