分离擦除与写入机制,提升线性注意力长文本处理能力
Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention
- 引入通道级擦除门和写入门,解耦旧内容清除与新内容写入
- 13亿参数模型在1000亿文本上训练,长上下文检索任务表现最优
- 特别适合需要精准记忆管理的长序列语言建模场景
线性注意力用固定大小的递归状态替代softmax注意力的无界缓存,将序列混合降至线性时间,解码内存恒定。难点在于如何在不破坏已有关联的前提下编辑压缩后的记忆。现有方法如Delta规则模型在写入前减去当前读取值,Kimi Delta Attention(KDA)通过通道级衰减强化遗忘。但主动编辑仍用单一标量门控制两个不同行为:键侧旧内容擦除量和值侧新内容写入量。本文提出Gated DeltaNet-2,继承自适应遗忘和通道级衰减的同时,解决这一共享局限——即擦除与写入的标量绑定。该模型采用通道级擦除门b_t和写入门w_t,分别控制两者,当两门退化为同一标量时还原为KDA,当衰减也退化时还原为Gated DeltaNet。推导出快速权重更新视角,设计带通道级衰减的分块WY算法,以及保留高效并行训练的门感知反向传播。在13亿参数、1000亿细粒度教育数据(FineWeb-Edu)token上训练的Gated DeltaNet-2,在语言建模、常识推理和检索任务中优于Mamba-2、Gated DeltaNet、KDA及Mamba-3变体。其优势在长上下文RULER针堆藏匿测试中尤为显著,多键检索设置表现更优,且在递归与混合架构下均保持强劲性能。
原文摘要 · Abstract (English)
Linear attention replaces the unbounded cache of softmax attention with a fixed-size recurrent state, reducing sequence mixing to linear time and decoding to constant memory. The hard part is not just what to forget, but how to edit this compressed memory without scrambling existing associations. Delta-rule models subtract the current read before writing a new value, and Kimi Delta Attention (KDA) sharpens forgetting with channel-wise decay. But the active edit still uses a single scalar gate to control two different things: how much old content to erase on the key side and how much new content to commit on the value side. We introduce Gated DeltaNet-2, which generalizes both Gated DeltaNet and KDA by inheriting adaptive forgetting and channel-wise decay while addressing their shared limitation, the scalar tie between erasing and writing. Gated Delta Rule-2 separates these roles with a channel-wise erase gate b_t and a channel-wise write gate w_t, reducing to KDA when both gates collapse to the same scalar and to Gated DeltaNet when the decay also collapses. We derive a fast-weight update view, a chunkwise WY algorithm with channel-wise decay absorbed into asymmetric erase factors, and a gate-aware backward pass that preserves efficient parallel training. At 1.3B parameters trained on 100B FineWeb-Edu tokens, Gated DeltaNet-2 achieves the strongest overall results among Mamba-2, Gated DeltaNet, KDA, and Mamba-3 variants across language modeling, commonsense reasoning, and retrieval. Its advantage is most pronounced on long-context RULER needle-in-a-haystack benchmarks, where it improves the evaluated multi-key retrieval setting and remains strong in both recurrent and hybrid settings. Code is available at https://github.com/NVlabs/GatedDeltaNet-2.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。