用傅里叶变换实现环卷积注意力，让Transformer处理长序列更快更省
CAT: Circular-Convolutional Attention for Sub-Quadratic Transformers
- 基于傅里叶变换的环卷积设计，将注意力复杂度降至O(NlogN)
- 在相同精度下比标准Transformer快约10%，参数更少
- 适合需要高效处理长序列的模型开发与部署
Transformer在自然语言处理和计算机视觉中取得显著突破，但其标准注意力机制仍存在O(N²)复杂度，限制了长序列的可扩展性。本文提出环卷积注意力（CAT），一种基于傅里叶变换的方法，通过高效应用环卷积降低计算复杂度，同时保持表征能力。CAT实现O(NlogN)计算量，减少全连接层的可学习参数，且不引入额外重操作，在原始PyTorch实现中获得约10%的速度提升，并保持一致的准确率。基于工程同构Transformer（EIT）框架，CAT不仅具备实际效率与易实现性，还为未来高性能Transformer架构的设计提供洞见。消融实验揭示了CAT成功的关键条件，为可扩展注意力机制提供了普适性指导。
原文摘要 · Abstract (English)
Transformers have driven remarkable breakthroughs in natural language processing and computer vision, yet their standard attention mechanism still imposes O(N^2) complexity, hindering scalability to longer sequences. We introduce Circular-convolutional ATtention (CAT), a Fourier-based approach that efficiently applies circular convolutions to reduce complexity without sacrificing representational power. CAT achieves O(NlogN) computations, requires fewer learnable parameters by streamlining fully connected layers, and introduces no additional heavy operations, resulting in consistent accuracy improvements and about a 10% speedup in naive PyTorch implementations. Based on the Engineering-Isomorphic Transformers (EITs) framework, CAT's design not only offers practical efficiency and ease of implementation, but also provides insights to guide the development of future high-performance Transformer architectures. Finally, our ablation studies highlight the key conditions underlying CAT's success, shedding light on broader principles for scalable attention mechanisms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。