用可高效适配的专家模块解决ViT搜索中的特征坍缩问题
TAS-LoRA: Transformer Architecture Search with Mixture-of-LoRA Experts
- 引入混合低秩专家机制，让子网络学习专属特征
- 在ImageNet上性能超越现有方法，多基准提升显著
- 适合追求高效自动架构设计的研究者与工程师
Transformer架构搜索（TAS）可自动发现最优视觉Transformer（ViT）结构，减少人工设计成本。但现有TAS方法存在特征坍缩问题：超网络中子网络因共享权重无法学习专属特征，制约其性能。为此，本文提出TAS-LoRA，引入参数高效的低秩适配（LoRA），实现子网络专属特征学习，同时保持计算效率。TAS-LoRA采用混合式LoRA专家（MoLE）策略，由轻量级路由器根据子网结构动态分配LoRA专家，并提出分组路由器初始化技术，促进早期专家间特征多样性。在ImageNet及多个迁移学习基准（包括CIFAR-10/100、Flowers、CARS、INAT-19）上的大量实验表明，TAS-LoRA有效缓解特征坍缩，显著提升性能，优于当前最先进TAS方法。
原文摘要 · Abstract (English)
Transformer architecture search (TAS) discovers optimal vision transformer (ViT) architectures automatically, reducing human effort to manually design ViTs. However, existing TAS methods suffer from the feature collapse problem, where subnets within a supernet fail to learn subnet-specific features, mainly due to the shared weights in a supernet, limiting the performance of individual subnets. To address this, we propose TAS-LoRA, a novel method that introduces parameter-efficient low-rank adaptation (LoRA) to enable subnet-specific feature learning, while maintaining computational efficiency. TAS-LoRA incorporates a Mixture-of-LoRAExperts (MoLE) strategy, where a lightweight router dynamically assigns LoRA experts based on subnet architectures, and introduces a group-wise router initialization technique to encourage diverse feature learning across experts early in training. Extensive experiments on ImageNet and several transfer learning benchmarks, including CIFAR-10/100, Flowers, CARS, and INAT-19, demonstrate that TAS-LoRA mitigates feature collapse effectively, improving performance over state-of-the-art TAS methods significantly.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。