提出新型注意力机制，用更少计算实现接近全注意力效果
Power-based Partial Attention: Bridging Linear-Complexity and Full Attention
- 设计幂次控制的局部注意力，复杂度介于线性和全注意力之间
- 实验发现当幂次参数小于1时，性能接近全注意力且计算量更低
- 适合追求高效推理的NLP模型部署场景
Transformer研究普遍认为‘注意力是全部所需’，但注意力所需量尚未系统量化。二次复杂度 $O(L^2)$ 的注意力是否必要？是否存在亚二次注意力机制可达到相近性能？为此，本文提出幂基部分注意力（PPA），其复杂度为 $O(L^{1+p})$，其中 $0 \≤ p \≤ 1$：当 $p=0$ 时对应线性复杂度滑动窗口注意力，$p=1$ 时对应全注意力。通过该机制，可系统分析注意力规模与模型性能的关系。实验显示性能随 $p$ 呈S型变化，在极小范围内从线性注意力过渡到全注意力，并在 $p \to 1$ 时趋于平稳。结果表明存在 $0 < p < 1$，使得 $O(L^{1+p})$ 注意力足以达到 $O(L^2)$ 全注意力的性能。
原文摘要 · Abstract (English)
It is widely accepted from transformer research that "attention is all we need", but the amount of attention required has never been systematically quantified. Is quadratic $O(L^2)$ attention necessary, or is there a sub-quadratic attention mechanism that can achieve comparable performance? To answer this question, we introduce power-based partial attention (PPA), an attention mechanism of order $O(L^{1+p})$, where $0 \leq p \leq 1$, such that $p=0$ corresponds to sliding window attention with linear complexity, and $p=1$ corresponds to full attention. With this attention construction, we can explore how transformer architecture performance varies as a function of the attention scaling behavior controlled by $p$. The overall trend from our experiments shows an S-curve-like behavior where the performance transitions from sliding-window (linear-complexity) attention to full attention over a narrow window of $p$ values, and plateaus as $p$ approaches $1$. In our experiments, we show that there exists $0<p<1$ such that $O(L^{1+p})$ attention is sufficient to achieve similar results as $O(L^2)$ full attention.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。