给视觉Transformer设计结构化初始化，小数据下表现接近CNN。
Structured Initialization for Vision Transformers
- 用仿卷积的结构化初始化替代随机初始化，不改架构
- 在小数据集上显著优于标准ViT初始化，大模型保持竞争力
- 适配Swin、MLP-Mixer等结构，适合数据少的视觉任务
卷积神经网络（CNN）具有强归纳偏置，在小规模数据上表现优异。本文提出将这种偏置融入视觉变压器（ViT），不通过结构改动，仅通过初始化实现。动机源于实验发现：随机脉冲滤波器在CNN中可达到与学习到的滤波器相当的性能。相比现有依赖经验性启发的ViT初始化策略（如使用预训练注意力权重或关注注意力分布但不强制结构），本方法显著提升性能。在多个小中规模基准测试中表现突出，包括Food-101、CIFAR-10、CIFAR-100、STL-10、Flowers和Pets；在ImageNet-1K等大规模数据集上也保持相当水平。该初始化策略可轻松集成至Swin Transformer、MLP-Mixer等多种Transformer架构，且持续带来性能提升。
原文摘要 · Abstract (English)
Convolutional Neural Networks (CNNs) inherently encode strong inductive biases, enabling effective generalization on small-scale datasets. In this paper, we propose integrating this inductive bias into ViTs, not through an architectural intervention but solely through initialization. The motivation here is to have a ViT that can enjoy strong CNN-like performance when data assets are small, but can still scale to ViT-like performance as the data expands. Our approach is motivated by our empirical results that random impulse filters can achieve commensurate performance to learned filters within a CNN. We improve upon current ViT initialization strategies, which typically rely on empirical heuristics such as using attention weights from pretrained models or focusing on the distribution of attention weights without enforcing structures. Empirical results demonstrate that our method significantly outperforms standard ViT initialization across numerous small and medium-scale benchmarks, including Food-101, CIFAR-10, CIFAR-100, STL-10, Flowers, and Pets, while maintaining comparative performance on large-scale datasets such as ImageNet-1K. Moreover, our initialization strategy can be easily integrated into various transformer-based architectures such as Swin Transformer and MLP-Mixer with consistent improvements in performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。