用可微搜索找到更高效的新版视觉Transformer架构
DASViT: Differentiable Architecture Search for Vision Transformer
- 提出可微搜索方法DASViT，自动优化ViT结构
- 新架构参数和计算量更少，性能超越ViT-B/16
- 适合追求高效视觉模型的研究者与工程师
设计有效的神经网络是深度学习的核心，神经网络架构搜索（NAS）已成为自动化这一过程的强大工具。在现有NAS方法中，可微架构搜索（DARTS）因其高效性和易用性而广受关注，推动了诸多进展。随着视觉变换器（ViT）的兴起，研究者将NAS应用于探索ViT架构，但多集中于宏观搜索空间，并依赖进化算法等离散方法。这些方法虽保证可靠性，却难以发现创新设计，需大量计算资源且耗时长。为此，我们提出面向视觉变换器的可微架构搜索（DASViT），填补了可微搜索在ViT领域的空白，发现了突破传统编码器结构的新设计。实验表明，DASViT生成的架构在多个数据集上优于ViT-B/16，且参数更少、计算量更低。
原文摘要 · Abstract (English)
Designing effective neural networks is a cornerstone of deep learning, and Neural Architecture Search (NAS) has emerged as a powerful tool for automating this process. Among the existing NAS approaches, Differentiable Architecture Search (DARTS) has gained prominence for its efficiency and ease of use, inspiring numerous advancements. Since the rise of Vision Transformers (ViT), researchers have applied NAS to explore ViT architectures, often focusing on macro-level search spaces and relying on discrete methods like evolutionary algorithms. While these methods ensure reliability, they face challenges in discovering innovative architectural designs, demand extensive computational resources, and are time-intensive. To address these limitations, we introduce Differentiable Architecture Search for Vision Transformer (DASViT), which bridges the gap in differentiable search for ViTs and uncovers novel designs. Experiments show that DASViT delivers architectures that break traditional Transformer encoder designs, outperform ViT-B/16 on multiple datasets, and achieve superior efficiency with fewer parameters and FLOPs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。