提出统一线性注意力模型，显著提升长序列建模效果。
MetaLA: Unified Optimal Linear Approximation to Softmax Attention Map
- 从理论出发统一现有线性注意力模型，推导最优设计条件。
- 在多任务测试中优于现有线性模型，尤其在长序列上表现突出。
- 适合追求高效长序列建模的NLP与视觉任务研究者。
多种线性复杂度模型（如LinFormer、SSM、LinRNN）被提出以替代Transformer中的传统softmax注意力机制，但其最优设计仍不明确。本文从理论角度寻找softmax注意力的最佳线性近似，将现有线性模型统一为线性注意力形式，并识别出最优设计的三个条件：动态记忆能力、静态近似能力、参数最少近似。发现当前模型均未满足全部条件，导致性能受限。为此，提出元线性注意力（MetaLA），满足所有条件。在多查询关联召回（MQAR）、语言建模、图像分类及长程竞技场（LRA）基准上的实验表明，MetaLA优于现有线性模型。
原文摘要 · Abstract (English)
Various linear complexity models, such as Linear Transformer (LinFormer), State Space Model (SSM), and Linear RNN (LinRNN), have been proposed to replace the conventional softmax attention in Transformer structures. However, the optimal design of these linear models is still an open question. In this work, we attempt to answer this question by finding the best linear approximation to softmax attention from a theoretical perspective. We start by unifying existing linear complexity models as the linear attention form and then identify three conditions for the optimal linear attention design: 1) Dynamic memory ability; 2) Static approximation ability; 3) Least parameter approximation. We find that none of the current linear models meet all three conditions, resulting in suboptimal performance. Instead, we propose Meta Linear Attention (MetaLA) as a solution that satisfies these conditions. Our experiments on Multi-Query Associative Recall (MQAR) task, language modeling, image classification, and Long-Range Arena (LRA) benchmark demonstrate that MetaLA is more effective than the existing linear models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。