小数据下优化微型ViT，提升性能且减少冗余。
Powerful Design of Small Vision Transformer on CIFAR10
- 用低秩压缩减少注意力冗余，损失小。
- 多分类标记提升全局表征，准确率提高。
- 适合做轻量级视觉模型的工程实践参考。
视觉变换器（ViT）在大规模数据集上表现优异，但在小数据集上的性能常不如卷积神经网络（CNN）。本文以CIFAR-10为基准，系统研究微型ViT在小数据下的设计与优化，评估数据增强、分块标记初始化、低秩压缩及多类别标记策略的影响。实验发现，多头潜在注意力（MLA）中查询的低秩压缩导致性能损失极小，表明ViT存在冗余；引入多个CLS标记可增强全局表征能力，显著提升准确率。这些发现为微型ViT的高效设计提供了完整框架，具有实际应用价值。代码已公开于 https://github.com/erow/PoorViTs。
原文摘要 · Abstract (English)
Vision Transformers (ViTs) have demonstrated remarkable success on large-scale datasets, but their performance on smaller datasets often falls short of convolutional neural networks (CNNs). This paper explores the design and optimization of Tiny ViTs for small datasets, using CIFAR-10 as a benchmark. We systematically evaluate the impact of data augmentation, patch token initialization, low-rank compression, and multi-class token strategies on model performance. Our experiments reveal that low-rank compression of queries in Multi-Head Latent Attention (MLA) incurs minimal performance loss, indicating redundancy in ViTs. Additionally, introducing multiple CLS tokens improves global representation capacity, boosting accuracy. These findings provide a comprehensive framework for optimizing Tiny ViTs, offering practical insights for efficient and effective designs. Code is available at https://github.com/erow/PoorViTs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。