arXiv:2412.14678cs.CV2024-12中稿 · AAAI

用非线性函数数量划分搜索空间,实现高效少样本神经网络架构搜索。

Efficient Few-Shot Neural Architecture Search by Counting the Number of Nonlinear Functions

  • 按非线性函数数量分组子网络,避免参数共享干扰。
  • 可降低每个超网通道数,支持多超网并行训练。
  • 适合资源受限场景下的快速架构搜索。

神经网络架构搜索(NAS)可自动从搜索空间中找到最优架构。现有方法通常使用包含所有可能架构的超网,但共享参数的子网特性差异会干扰训练。为此,少样本NAS方法将搜索空间划分为若干子空间,并为每个子空间分配独立超网,以减少权重共享,虽性能领先但计算开销大。本文提出一种新方法,基于非线性函数的数量划分搜索空间,使每个子空间内子网具有相同数量的非线性函数。该划分无需比较超网梯度,效率更高。同时发现,划分后可显著降低每个超网的通道维度,便于高效训练多个超网。此外,引入超网均衡采样(SBS)策略,在每轮训练中均匀采样多个子网,确保在有限训练步数内各超网均衡学习。大量实验验证了该方法的有效性。代码已公开于https://cvlab.yonsei.ac.kr/projects/EFS-NAS。

原文摘要 · Abstract (English)

Neural architecture search (NAS) enables finding the best-performing architecture from a search space automatically. Most NAS methods exploit an over-parameterized network (i.e., a supernet) containing all possible architectures (i.e., subnets) in the search space. However, the subnets that share the same set of parameters are likely to have different characteristics, interfering with each other during training. To address this, few-shot NAS methods have been proposed that divide the space into a few subspaces and employ a separate supernet for each subspace to limit the extent of weight sharing. They achieve state-of-the-art performance, but the computational cost increases accordingly. We introduce in this paper a novel few-shot NAS method that exploits the number of nonlinear functions to split the search space. To be specific, our method divides the space such that each subspace consists of subnets with the same number of nonlinear functions. Our splitting criterion is efficient, since it does not require comparing gradients of a supernet to split the space. In addition, we have found that dividing the space allows us to reduce the channel dimensions required for each supernet, which enables training multiple supernets in an efficient manner. We also introduce a supernet-balanced sampling (SBS) technique, sampling several subnets at each training step, to train different supernets evenly within a limited number of training steps. Extensive experiments on standard NAS benchmarks demonstrate the effectiveness of our approach. Our code is available at https://cvlab.yonsei.ac.kr/projects/EFS-NAS.

架构搜索少样本超网高效

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。