用聚类替代注意力，让图像描述更快更准。
Beyond Self-Attention: Sub-Quadratic Vision Transformers for Fast Image Captioning
- 用高斯混合模型聚类图像块，替代传统自注意力
- 计算复杂度从O(n²)降到O(nK)，K远小于n
- 在Flickr 30K上表现优于现有方法，适合高效生成
图像字幕生成旨在为给定图像生成连贯且语义丰富的文本描述。尽管基于Transformer的架构取得了显著进展，但现有方法仍面临局部特征表达不足和二次方自注意力计算开销大的问题。本文提出一种重构视觉Transformer架构的新方法：将标准自注意力替换为基于高斯混合模型（GMM）的概率性注意力机制，该机制通过期望最大化（EM）算法将相似图像块聚成固定数量的簇，从而将计算复杂度从二次方O(n²)降低至线性O(nK)，其中K << n。使用基于GPT的自回归解码器生成字幕。在Flickr 30K数据集上的实验表明，该模型在保持高质量生成的同时实现了显著的效率提升，性能优于现有方法。
原文摘要 · Abstract (English)
Image captioning is a challenging and significant task that aims to generate coherent and semantically meaningful textual descriptions for given images. To accomplish this task, it requires a deep understanding of visual content along with the ability to express that understanding in natural language. Despite remarkable progress with transformer-based architectures, existing approaches often suffer from limitations, such as a lack of rich local feature representations and the high computational cost of quadratic self-attention. The proposed model focuses on improving computational efficiency by restructuring the vision transformer architecture. In designing this approach, the standard self-attention mechanism in Vision Transformers is replaced with a probabilistic transformer approach based on a Gaussian Mixture Model (GMM), a soft-clustering technique. Instead of computing pairwise attention among all image patches, the model groups similar patches into a fixed number of clusters using an Expectation-Maximization (EM) algorithm. This clustering-based mechanism reduces the computational complexity from quadratic O(n^2) to linear O(nK), where K << n. The autoregressive GPT-based decoder is used for caption generation. The model is evaluated on the Flickr 30K dataset, demonstrating competitive and significant improvement over existing works.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。