用孪生网络预测架构优劣关系,大幅降低多目标神经网络搜索成本。
SiamNAS: Siamese Surrogate Model for Dominance Relation Prediction in Multi-objective Neural Architecture Search
- 设计孪生网络代理模型,预测候选架构的支配关系。
- 在NAS-Bench-201上仅用0.01 GPU天找到最优与次优架构。
- 适合追求高效多目标神经架构搜索的研究者使用。
现代神经网络架构搜索(NAS)本质上是多目标问题,需在准确率、参数量和计算成本之间权衡。这一复杂性导致搜索过程计算开销巨大,难以有效求解。为此,我们提出一种新型代理建模方法,利用一组孪生网络模块预测候选架构间的支配关系。该代理模型轻量且易于训练,准确率达92%,并用基于模型规模的启发式规则替代了生存选择策略中的拥挤度计算。集成至SiamNAS框架后,该设计在搜索过程中完全避免了昂贵的评估。在NAS-Bench-201上的实验表明,该框架能以极低计算成本识别帕累托最优解。所提SiamNAS在0.01 GPU天内找到了CIFAR-10上测试误差最低的架构,以及ImageNet上第二低的架构。本研究证明了孪生网络代理模型在多任务优化中的泛化潜力,支持跨任务的联合优化,并为生成帕累托集集合(SOS)提供了可能,适用于多样化任务场景。
原文摘要 · Abstract (English)
Modern neural architecture search (NAS) is inherently multi-objective, balancing trade-offs such as accuracy, parameter count, and computational cost. This complexity makes NAS computationally expensive and nearly impossible to solve without efficient approximations. To address this, we propose a novel surrogate modelling approach that leverages an ensemble of Siamese network blocks to predict dominance relationships between candidate architectures. Lightweight and easy to train, the surrogate achieves 92% accuracy and replaces the crowding distance calculation in the survivor selection strategy with a heuristic rule based on model size. Integrated into a framework termed SiamNAS, this design eliminates costly evaluations during the search process. Experiments on NAS-Bench-201 demonstrate the framework's ability to identify Pareto-optimal solutions with significantly reduced computational costs. The proposed SiamNAS identified a final non-dominated set containing the best architecture in NAS-Bench-201 for CIFAR-10 and the second-best for ImageNet, in terms of test error rate, within 0.01 GPU days. This proof-of-concept study highlights the potential of the proposed Siamese network surrogate model to generalise to multi-tasking optimisation, enabling simultaneous optimisation across tasks. Additionally, it offers opportunities to extend the approach for generating Sets of Pareto Sets (SOS), providing diverse Pareto-optimal solutions for heterogeneous task settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。