arXiv:2503.22733cs.LG2025-03被引 10

不训练即可高效搜索最优网络结构,准确率领先。

RBFleX-NAS: Training-Free Neural Architecture Search Using Radial Basis Function Kernel and Hyperparameter Detection

  • 用径向基函数核融合输入特征与激活输出进行评分
  • 在NAS-Bench-201上实现更高顶1准确率,搜索时间短
  • 适合需要快速探索激活函数的模型设计场景

神经架构搜索(NAS)是一种自动化设计最优神经网络架构的技术。传统方法需大量训练来评估候选网络,耗时且资源密集。为此,训练自由的NAS被提出以缩短搜索时间。然而,现有先进方法难以精准区分优劣网络,导致性能预测不准,最终影响顶1准确率。此外,在激活函数探索方面表现不佳。本文提出RBFleX-NAS,一种新型训练自由NAS框架,利用径向基函数(RBF)核同时考虑最后一层的激活输出与输入特征。我们还设计了一种检测算法,基于激活输出与输入特征图识别最优超参数。在多个NAS基准上验证了该方法的有效性。RBFleX-NAS在NAS-Bench-201和NAS-Bench-SSS上显著优于现有训练自由方法,顶1准确率更高,搜索时间短。其肯德尔相关系数也高于基于层的训练自由方法。此外,我们提出NAFBee新激活函数设计空间,扩展常用激活函数类型。在此空间中,RBFleX-NAS能准确识别最佳网络,展现出明显优势。

原文摘要 · Abstract (English)

Neural Architecture Search (NAS) is an automated technique to design optimal neural network architectures for a specific workload. Conventionally, evaluating candidate networks in NAS involves extensive training, which requires significant time and computational resources. To address this, training-free NAS has been proposed to expedite network evaluation with minimal search time. However, state-of-the-art training-free NAS algorithms struggle to precisely distinguish well-performing networks from poorly-performing networks, resulting in inaccurate performance predictions and consequently sub-optimal top-1 network accuracy. Moreover, they are less effective in activation function exploration. To tackle the challenges, this paper proposes RBFleX-NAS, a novel training-free NAS framework that accounts for both activation outputs and input features of the last layer with a Radial Basis Function (RBF) kernel. We also present a detection algorithm to identify optimal hyperparameters using the obtained activation outputs and input feature maps. We verify the efficacy of RBFleX-NAS over a variety of NAS benchmarks. RBFleX-NAS significantly outperforms state-of-the-art training-free NAS methods in terms of top-1 accuracy, achieving this with short search time in NAS-Bench-201 and NAS-Bench-SSS. In addition, it demonstrates higher Kendall correlation compared to layer-based training-free NAS algorithms. Furthermore, we propose NAFBee, a new activation design space that extends the activation type to encompass various commonly used functions. In this extended design space, RBFleX-NAS demonstrates its superiority by accurately identifying the best-performing network during activation function search, providing a significant advantage over other NAS algorithms.

神经架构搜索训练自由激活函数

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。