通过引入归纳偏置，让视觉Transformer在小数据上表现更好
IBiT: Utilizing Inductive Biases to Create a More Data Efficient Attention Mechanism
- 用可学习掩码引入卷积网络的归纳偏置
- 小数据集上准确率显著提升，无需知识蒸馏
- 适合数据稀缺场景的视觉模型研究者
近年来，基于Transformer的架构已成为计算机视觉应用的主流。尽管Transformer具有可解释性且随数据规模扩大而表现良好，但缺乏卷积神经网络的归纳偏置。虽然这些偏置可在大规模数据上被学习到，但我们证明通过引入可学习掩码来施加归纳偏置，能使视觉Transformer在小数据集上高效学习，无需知识蒸馏。这类模型被称为归纳偏置图像Transformer（IBiT），在小数据集上表现更优，同时保持了Transformer的可解释性。
原文摘要 · Abstract (English)
In recent years, Transformer-based architectures have become the dominant method for Computer Vision applications. While Transformers are explainable and scale well with dataset size, they lack the inductive biases of Convolutional Neural Networks. While these biases may be learned on large datasets, we show that introducing these inductive biases through learned masks allow Vision Transformers to learn on much smaller datasets without Knowledge Distillation. These Transformers, which we call Inductively Biased Image Transformers (IBiT), are significantly more accurate on small datasets, while retaining the explainability Transformers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。