用L1距离和卷积替代注意力中的乘法，大幅降低能耗。
EcoTransformer: Attention without Multiplication
- 用L1距离与拉普拉斯核卷积计算注意力，无需矩阵乘法
- 在多个任务上性能媲美或超过传统注意力，能耗显著降低
- 适合追求低功耗、高效推理的NLP与多模态应用
Transformer凭借其缩放点积注意力机制已成为现代人工智能的基础架构。然而，该机制计算密集，能耗高昂。本文提出新型Transformer架构EcoTransformer，将输出上下文向量构建为值向量的卷积，使用拉普拉斯核，距离度量采用查询与键之间的L1范数。相比基于点积的注意力，新方法无需矩阵乘法即可计算注意力得分。在自然语言处理、生物信息学和视觉任务中，其性能达到或超越传统缩放点积注意力，同时能耗显著降低。
原文摘要 · Abstract (English)
The Transformer, with its scaled dot-product attention mechanism, has become a foundational architecture in modern AI. However, this mechanism is computationally intensive and incurs substantial energy costs. We propose a new Transformer architecture EcoTransformer, in which the output context vector is constructed as the convolution of the values using a Laplacian kernel, where the distances are measured by the L1 metric between the queries and keys. Compared to dot-product based attention, the new attention score calculation is free of matrix multiplication. It performs on par with, or even surpasses, scaled dot-product attention in NLP, bioinformatics, and vision tasks, while consuming significantly less energy. (This version (v2) supersedes v1 and reflects the intended release and licensing.)
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。