让注意力机制按需调整窗口大小，提升长距离依赖捕捉能力
MSWA: Refining Local Attention with Multi-ScaleWindow Attention
- 不同头和层采用可变窗口大小，动态适应上下文尺度
- 在语言建模与常识推理任务中，性能超越传统局部注意力
- 适合需要高效处理长文本的模型设计者
基于Transformer的大型语言模型在众多自然语言任务中表现卓越。然而，标准自注意力机制存在时间复杂度为二次方、缓存大小线性增长的问题。滑动窗口注意力（SWA）通过限制注意力范围至固定大小的局部上下文窗口来缓解该问题。但SWA在每层每个头中使用统一窗口大小，难以有效捕捉不同尺度的上下文信息。为此，我们提出多尺度窗口注意力（MSWA），在Transformer中实现跨头和跨层的多样化窗口大小分配。它不仅允许同一层内不同头采用不同窗口大小，还从浅层到深层逐步增大窗口尺寸，从而支持对不同长度和距离的上下文信息进行捕捉。在语言建模和常识推理任务上的实验结果表明，MSWA在有效性与效率方面均优于传统局部注意力。
原文摘要 · Abstract (English)
Transformer-based LLMs have achieved exceptional performance across a wide range of NLP tasks. However, the standard self-attention mechanism suffers from quadratic time complexity and linearly increased cache size. Sliding window attention (SWA) solves this problem by restricting the attention range to a fixed-size local context window. Nevertheless, SWA employs a uniform window size for each head in each layer, making it inefficient in capturing context of varying scales. To mitigate this limitation, we propose Multi-Scale Window Attention (MSWA) which applies diverse window sizes across heads and layers in the Transformer. It not only allows for different window sizes among heads within the same layer but also progressively increases window size allocation from shallow to deep layers, thus enabling the model to capture contextual information with different lengths and distances. Experimental results on language modeling and common-sense reasoning tasks substantiate that MSWA outperforms traditional local attention in both effectiveness and efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。