跨数据集搜索可迁移组件，高效构建不同大小的模型。
Learngene Search Across Multiple Datasets for Building Variable-Sized Models
- 构建多数据集可搜索超网络，自动优选最优组件路径。
- 在多个数据集上性能接近预训练微调，但存储与训练成本大幅降低。
- 适合资源受限场景下快速部署多样规模模型的研究者。
深度学习在多种资源约束下广泛应用，催生了如Vision Transformer（ViT）系列等不同规模的模型。部署这些模型通常需耗费大量成本进行预训练和微调。Learngene范式通过从预训练祖先模型（Ans-Net）中提取可迁移组件（称为learngenes），用于初始化不同规模的后代模型（Des-Nets）。现有learngene提取方法依赖单一数据集，限制下游性能。为此，本文提出跨多数据集的Learngene搜索方法（LSAMD），将Ans-Net扩展为包含数据集特定块和数据集适配器（DADs）的可搜索超网络。训练过程中，LSAMD为每个数据集搜索最优架构路径，将被频繁选中的基础模块提取为learngenes以初始化Des-Nets。在多个数据集上的实验表明，LSAMD在性能上可媲美预训练-微调方法，同时显著降低存储和训练开销。
原文摘要 · Abstract (English)
Deep learning methods are widely used under diverse resource constraints, resulting in models of varying sizes, such as the Vision Transformer (ViT) series. Deploying these models typically requires costly pretraining and finetuning. The Learngene paradigm addresses this issue by extracting transferable components, called learngenes, from a pretrained ancestry model (Ans-Net) to initialize variable-sized descendant models (Des-Nets).Existing learngene extraction methods rely on a single dataset, limiting downstream performance. To address this limitation, we propose Learngene Search Across Multiple Datasets for Building Variable-Sized Models (LSAMD). LSAMD expands the Ans-Net into a searchable super Ans-Net with dataset-specific blocks and dataset adapters (DADs). During training, LSAMD searches for an optimal architecture path for each dataset. The base blocks most frequently selected across datasets are extracted as learngenes for initializing Des-Nets.Experiments on multiple datasets show that LSAMD achieves performance comparable to pretrain-finetune methods while significantly reducing storage and training costs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。