动态调整子网训练策略，提升神经网络搜索公平性与精度
Subnet-Aware Dynamic Supernet Training for Neural Architecture Search
- 根据子网复杂度自适应调节学习率衰减，解决训练偏倚问题
- 按结构特征分组管理动量，减少训练噪声，提升稳定性
- 兼容主流NAS方法，无需额外计算开销，效果显著
N-shot神经架构搜索（NAS）通过包含所有候选子网的超网络进行搜索。传统方法对所有子网采用固定的训练策略（如统一的学习率调度器和优化器），但未考虑子网间的差异，导致两个问题：(1) 超网络训练偏向低复杂度子网（不公平）；(2) 动量更新存在噪声（噪声动量）。本文提出一种动态超网络训练方法，通过自适应调整训练策略来缓解上述问题。具体地，引入复杂度感知学习率调度器（CaLR），根据子网复杂度动态调节学习率衰减速率，缓解不公平现象；同时提出动量分离技术（MS），将结构相似的子网分组并为每组使用独立动量，避免动量噪声。该方法可无缝集成至多种N-shot NAS方法，计算开销极小，却显著提升搜索性能。在多个搜索空间（如NAS-Bench-201、Mobilenet）和数据集（如CIFAR-10/100、ImageNet）上均验证了其有效性。
原文摘要 · Abstract (English)
N-shot neural architecture search (NAS) exploits a supernet containing all candidate subnets for a given search space. The subnets are typically trained with a static training strategy (e.g., using the same learning rate (LR) scheduler and optimizer for all subnets). This, however, does not consider that individual subnets have distinct characteristics, leading to two problems: (1) The supernet training is biased towards the low-complexity subnets (unfairness); (2) the momentum update in the supernet is noisy (noisy momentum). We present a dynamic supernet training technique to address these problems by adjusting the training strategy adaptive to the subnets. Specifically, we introduce a complexity-aware LR scheduler (CaLR) that controls the decay ratio of LR adaptive to the complexities of subnets, which alleviates the unfairness problem. We also present a momentum separation technique (MS). It groups the subnets with similar structural characteristics and uses a separate momentum for each group, avoiding the noisy momentum problem. Our approach can be applicable to various N-shot NAS methods with marginal cost, while improving the search performance drastically. We validate the effectiveness of our approach on various search spaces (e.g., NAS-Bench-201, Mobilenet spaces) and datasets (e.g., CIFAR-10/100, ImageNet).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。