线性注意力让多模态模型更快更省资源，还能保持高性能。
On The Application of Linear Attention in Multimodal Transformers
- 用线性注意力替代传统注意力，计算量从二次方降为线性。
- 在多个ViT模型上测试，零样本准确率与原模型相当。
- 适合需要高效处理大规模多模态数据的研究者使用。
多模态Transformer是当前视觉-语言模型的核心架构，但其注意力机制的二次复杂度严重制约了可扩展性。本文探索线性注意力（Linear Attention, LA）在多模态框架中的可行性。通过引入LA，将计算开销从序列长度的二次方降低至线性，同时保持竞争力性能。我们在ViT-S/16、ViT-B/16和ViT-L/16架构上进行实验，训练数据来自LAION-400M，验证目标为ImageNet-21K上的零样本分类准确率。系统评估表明，线性注意力不仅显著降低计算成本，且遵循与标准softmax注意力相同的缩放规律。这些发现使线性注意力成为下一代需处理大规模复杂数据的多模态Transformer的稳健可扩展解决方案。
原文摘要 · Abstract (English)
Multimodal Transformers serve as the backbone for state-of-the-art vision-language models, yet their quadratic attention complexity remains a critical barrier to scalability. In this work, we investigate the viability of Linear Attention (LA) as a high-efficiency alternative within multimodal frameworks. By integrating LA, we reduce the computational overhead from quadratic to linear relative to sequence length while preserving competitive performance. We evaluate our approach across ViT-S/16, ViT-B/16, and ViT-L/16 architectures trained on the LAION-400M dataset, with validation focused on ImageNet-21K zero-shot accuracy. Our systematic evaluation demonstrates that Linear Attention not only yields significant computational savings but also adheres to the same scaling laws as standard softmax attention. These findings position Linear Attention as a robust, scalable solution for next-generation multimodal Transformers tasked with processing increasingly large and complex datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。