从小型子网逐步扩展，提升ViT搜索效率与精度
GrowTAS: Progressive Expansion from Small to Large Subnets for Efficient ViT Architecture Search
- 先训练小模型再逐步加入大模型，减少权重干扰
- 在ImageNet上准确率比现有方法最高提升1.8%
- 适合追求高效ViT架构自动设计的研究者
Transformer架构搜索（TAS）旨在自动发现高效的视觉变换器（ViTs），减少人工设计需求。现有TAS方法通常训练一个包含所有候选架构的超网络（supernet），但所有子网共享同一组权重，导致干扰严重，尤其影响小型子网性能。我们发现，经过良好训练的小型子网可作为训练大型子网的良好基础。受此启发，我们提出一种渐进式训练框架GrowTAS，从训练小型子网开始，逐步引入更大子网，有效降低干扰并稳定训练过程。此外，我们还提出GrowTAS+，仅微调部分权重以进一步提升大子网性能。在ImageNet及多个迁移学习基准（包括CIFAR-10/100、Flowers、CARS、INAT-19）上的大量实验表明，该方法优于当前主流TAS方法。
原文摘要 · Abstract (English)
Transformer architecture search (TAS) aims to automatically discover efficient vision transformers (ViTs), reducing the need for manual design. Existing TAS methods typically train an over-parameterized network (i.e., a supernet) that encompasses all candidate architectures (i.e., subnets). However, all subnets share the same set of weights, which leads to interference that degrades the smaller subnets severely. We have found that well-trained small subnets can serve as a good foundation for training larger ones. Motivated by this, we propose a progressive training framework, dubbed GrowTAS, that begins with training small subnets and incorporate larger ones gradually. This enables reducing the interference and stabilizing a training process. We also introduce GrowTAS+ that fine-tunes a subset of weights only to further enhance the performance of large subnets. Extensive experiments on ImageNet and several transfer learning benchmarks, including CIFAR-10/100, Flowers, CARS, and INAT-19, demonstrate the effectiveness of our approach over current TAS methods
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。