SWH用双流架构实现长序列建模，兼顾效率与精度。
Spectral-Window Hybrid (SWH)
- 双流并行：全局用卷积定理加速，局部用滑动窗口注意力
- 长序列时计算量从O(T²)降至O(T log T)，仍保持低困惑度
- 适合超长上下文任务，如长文本生成、超长时序预测
将序列建模扩展到超长上下文需要在计算效率与表征表达力之间取得平衡。尽管Transformer通过注意力机制实现了精准检索，但其O(T²)的复杂度限制了其在长时序任务中的应用。本文提出谱窗混合（Spectral-Window Hybrid, SWH）架构，将序列建模分解为两个并行分支：全局分支利用卷积定理在O(T log T)时间内建模长程衰减动态；局部分支采用滑动窗口注意力捕捉有限上下文内的词元交互。通过融合两路表示，SWH避免了全局注意力的计算瓶颈，同时保留局部精度。实验表明，SWH在短上下文上匹配标准Transformer的困惑度，并可高效线性扩展至长序列。代码已开源。
原文摘要 · Abstract (English)
Scaling sequence modeling to extreme contexts requires balancing computational efficiency with representational expressivity. While Transformers provide precise retrieval via the attention mechanism, their quadratic $\mathcal{O}(T^2)$ complexity limits their application to long-horizon tasks. In this work, we propose the \textbf{Spectral-Window Hybrid (SWH)}, an architecture that decouples sequence modeling into two \textit{parallel} streams: a global branch utilizing the Convolution Theorem to model long-range decay dynamics in $\mathcal{O}(T \log T)$ time, and a local branch employing sliding-window attention for token interactions within a bounded context. By aggregating these representations, SWH avoids the computational bottleneck of global attention while retaining local precision. We demonstrate that SWH matches the perplexity of standard Transformers on short contexts while enabling efficient linear scaling to extended sequences. The code is available at https://github.com/VladimerKhasia/SWH
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。