提出基于操作强度的架构选择方法，解决DARTS的退化问题。
OStr-DARTS: Differentiable Neural Architecture Search based on Operation Strength
- 用操作对损失的影响评估重要性，替代传统大小选择法。
- 在NAS-Bench-201和DARTS搜索空间上均实现更高性能架构。
- 无需修改优化过程，直接提升DARTS稳定性，适合高效架构搜索者。
可微分神经架构搜索（DARTS）已成为高效的神经架构搜索技术，其流程包含两步：首先通过梯度下降优化由混合操作构成的DARTS超网；其次根据贡献度选出最优操作构建最终架构。尽管提升了搜索效率，DARTS仍存在著名的退化问题，导致性能下降。现有研究多归因于超网优化失败，却较少关注选择策略。本文摒弃广泛使用的基于大小的选择方法，提出一种基于操作强度的新准则，通过操作对最终损失的影响来估计其重要性。实验表明，仅使用该准则即可有效缓解退化问题，且无需修改超网优化过程，说明基于大小的选择可能是DARTS不稳定的主因。在NAS-Bench-201与DARTS搜索空间上的实验验证了该方法的有效性。
原文摘要 · Abstract (English)
Differentiable architecture search (DARTS) has emerged as a promising technique for effective neural architecture search, and it mainly contains two steps to find the high-performance architecture: First, the DARTS supernet that consists of mixed operations will be optimized via gradient descent. Second, the final architecture will be built by the selected operations that contribute the most to the supernet. Although DARTS improves the efficiency of NAS, it suffers from the well-known degeneration issue which can lead to deteriorating architectures. Existing works mainly attribute the degeneration issue to the failure of its supernet optimization, while little attention has been paid to the selection method. In this paper, we cease to apply the widely-used magnitude-based selection method and propose a novel criterion based on operation strength that estimates the importance of an operation by its effect on the final loss. We show that the degeneration issue can be effectively addressed by using the proposed criterion without any modification of supernet optimization, indicating that the magnitude-based selection method can be a critical reason for the instability of DARTS. The experiments on NAS-Bench-201 and DARTS search spaces show the effectiveness of our method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。