通过门控与增量更新结合，提升长文本理解能力
Gated Delta Networks: Improving Mamba2 with Delta Rule
- 融合门控机制与增量更新规则，实现精准记忆控制
- 在多项任务中超越Mamba2和DeltaNet，长文本表现更优
- 适合需要高效处理长序列的自然语言应用
线性Transformer作为标准Transformer的高效替代，但在检索和长上下文任务中的表现受限。为此，近期研究探索了两种不同机制：门控用于自适应记忆控制，增量更新规则用于精确记忆修改。我们发现这两种机制具有互补性：门控可快速擦除记忆，而增量规则支持定向更新。基于此洞察，我们提出门控增量规则，并设计了适配现代硬件的并行训练算法。所提出的Gated DeltaNet架构在多个基准测试中持续优于Mamba2和DeltaNet，涵盖语言建模、常识推理、上下文检索、长度外推及长上下文理解。此外，通过构建混合架构，将Gated DeltaNet层与滑动窗口注意力或Mamba2层结合，实现了更高的训练效率和更优的任务性能。
原文摘要 · Abstract (English)
Linear Transformers have gained attention as efficient alternatives to standard Transformers, but their performance in retrieval and long-context tasks has been limited. To address these limitations, recent work has explored two distinct mechanisms: gating for adaptive memory control and the delta update rule for precise memory modifications. We observe that these mechanisms are complementary: gating enables rapid memory erasure while the delta rule facilitates targeted updates. Building on this insight, we introduce the gated delta rule and develop a parallel training algorithm optimized for modern hardware. Our proposed architecture, Gated DeltaNet, consistently surpasses existing models like Mamba2 and DeltaNet across multiple benchmarks, including language modeling, common-sense reasoning, in-context retrieval, length extrapolation, and long-context understanding. We further enhance performance by developing hybrid architectures that combine Gated DeltaNet layers with sliding window attention or Mamba2 layers, achieving both improved training efficiency and superior task performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。