提出滑动窗口递推机制，显著提升长序列建模速度。
Sliding Window Recurrences for Sequence Models
- 将递推计算按硬件内存层级分块，形成自然锯齿状窗口。
- 在10亿参数模型中，上下文达4K至32K时速度提升10%-40%。
- 可直接替换窗口注意力或线性递推，适合长文本生成任务。
多混合架构因更优的性能与质量，正成为语言建模的主流。本文提出一种分层分解框架，用于线性递推计算，使算法适配GPU内存层级，实现滑动窗口递推（Sliding Window Recurrences）。我们聚焦于将递推截断为与硬件对齐的窗口，这些窗口天然呈锯齿状，有效减少跨线程组通信开销。基于SWR，我们设计了Phalanx层，可作为窗口注意力或线性递推的即插即用替代方案。在参数量达10亿的多混合模型中，当上下文长度从4K扩展至32K时，Phalanx相较优化后的Transformer实现超过10%-40%的速度提升，同时保持相近的困惑度（perplexity）。
原文摘要 · Abstract (English)
Multi-hybrid architectures are poised to take over language modeling due to better quality and performance. We introduce a hierarchical decomposition framework for linear recurrences that allows us to develop algorithms aligned with GPU memory hierarchies, yielding Sliding Window Recurrences. We focus specifically on truncating recurrences to hardware-aligned windows which are naturally jagged, limiting costly inter-warp communication. Using SWR, we develop Phalanx layers that serve as drop-in replacements for windowed attention or linear recurrences. In 1B parameter multi-hybrid models, Phalanx achieves over 10-40% speedup across 4K to 32K context length over optimized Transformers while matching perplexity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。