用程序生成数据预训练视觉Transformer，提升模型效率与性能。
Can You Learn to See Without Images? Procedural Warm-Up for Vision Transformers
- 用算法生成无图像内容的数据，让ViT学习抽象计算先验。
- 仅用1%程序数据预训练，ImageNet-1K准确率提升超1.7%。
- 适合追求数据高效与跨域泛化的模型研究者。
Transformer模型具有高度通用性，暗示其在多种模态中存在有益的通用归纳偏置。本文探索通过在无视觉或语义内容的程序生成数据上预训练视觉Transformer（ViTs），来引入此类偏置。我们使用形式语法等简单算法生成数据，其结果与自然或合成图像无关。利用这些程序数据对ViTs进行预训练，跳过视觉补丁嵌入机制，促使模型内化抽象计算先验。随后进行标准图像训练时，该预热阶段显著提升数据效率、收敛速度和下游性能。例如，在ImageNet-1K上，仅分配1%训练预算给程序数据，即可使最终准确率提升超过1.7%。从性能影响看，1%程序数据的效果相当于ImageNet-1K数据的28%。
原文摘要 · Abstract (English)
Transformers are remarkably versatile, suggesting the existence of generic inductive biases beneficial across modalities. In this work, we explore a new way to instil such biases in vision transformers (ViTs) through pretraining on procedurally generated data devoid of visual or semantic content. We generate this data with simple algorithms such as formal grammars, so the results bear no relationship to either natural or synthetic images. We use this procedurally generated data to pretrain ViTs in a warm-up phase that bypasses their visual patch embedding mechanisms, thus encouraging the models to internalise abstract computational priors. When followed by standard image-based training, this warm-up significantly improves data efficiency, convergence speed, and downstream performance. On ImageNet-1K, for example, allocating just 1% of the training budget to procedural data improves final accuracy by over 1.7%. In terms of its effect on performance, 1% procedurally generated data is thus equivalent to 28% of the ImageNet-1K data. These findings suggest a promising path toward new data-efficient and domain-agnostic pretraining strategies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。