提出新型线性注意力机制，提升大模型视频生成效率
HLA: Hadamard Linear Attention
- 在相似度计算后统一引入非线性，而非分别作用于查询键
- 理论分析表明可逼近更高阶的Softmax函数，表达能力更强
- 无需复杂张量重塑，适合超大规模令牌的视频生成任务
注意力机制是Transformer成功的关键，依赖于对所有词元对之间的关系建模。为降低标准二次注意力的高计算开销，线性注意力被提出作为高效近似方法，通过核函数独立作用于输入后再计算成对相似性，实现高效计算，但其本质是低阶有理函数近似Softmax。本文提出哈达玛线性注意力（HLA），不同于以往方法，非线性不分别应用于查询和键，而是类似标准Softmax注意力，在成对相似性计算后引入，使非线性等价于更高阶有理函数以更优逼近Softmax。我们推导出一种与标准线性注意力类似的高效计算方案，且无需耗时的张量重塑操作。该方法在大型扩散Transformer模型用于视频生成的任务中验证了有效性，该任务涉及大量词元。
原文摘要 · Abstract (English)
The attention mechanism is an important reason for the success of transformers. It relies on computing pairwise relations between tokens. To reduce the high computational cost of standard quadratic attention, linear attention has been proposed as an efficient approximation. It employs kernel functions that are applied independently to the inputs before the pairwise similarities are calculated. That allows for an efficient computational procedure which, however, amounts to a low-degree rational function approximating softmax. We propose Hadamard Linear Attention (HLA). Unlike previous works on linear attention, the nonlinearity in HLA is not applied separately to queries and keys, but, analogously to standard softmax attention, after the pairwise similarities have been computed. It will be shown that the proposed nonlinearity amounts to a higher-degree rational function to approximate softmax. An efficient computational scheme for the proposed method is derived that is similar to that of standard linear attention. In contrast to other approaches, no time-consuming tensor reshaping is necessary to apply the proposed algorithm. The effectiveness of the approach is demonstrated by applying it to a large diffusion transformer model for video generation, an application that involves very large amounts of tokens.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。