用超注意力提升小型卷积Transformer，精度更高且更省资源
Enhancing compact convolutional transformers with super attention
- 引入令牌混合与序列池化，结合卷积令牌化器设计新架构
- 在CIFAR100上准确率提升至Top-1 46.29%、Top-5 76.31%
- 无需数据增强或学习率调度，训练稳定且推理更高效
本文提出一种视觉模型，采用令牌混合、序列池化和卷积令牌化器，在固定上下文长度任务中实现领先性能与高效推理。在CIFAR100基准测试中，模型将基线的Top-1和Top-5验证准确率从36.50%提升至46.29%，从66.33%提升至76.31%。当上下文长度小于嵌入维度时，其效率优于标准点积注意力（SDPA）Transformer，且模型尺寸仅为后者的60%。此外，该架构表现出高训练稳定性，无需依赖mixup、位置编码或学习率调度等技巧。代码已开源。
原文摘要 · Abstract (English)
In this paper, we propose a vision model that adopts token mixing, sequence-pooling, and convolutional tokenizers to achieve state-of-the-art performance and efficient inference in fixed context-length tasks. In the CIFAR100 benchmark, our model significantly improves the baseline of the top 1% and top 5% validation accuracy from 36.50% to 46.29% and 66.33% to 76.31%, while being more efficient than the Scaled Dot Product Attention (SDPA) transformers when the context length is less than the embedding dimension and only 60% the size. In addition, the architecture demonstrates high training stability and does not rely on techniques such as data augmentation like mixup, positional embeddings, or learning rate scheduling. We make our code available on Github.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。