用平滑激活函数解决架构搜索中的捷径主导问题，提升模型性能与泛化能力。
Regularizing Differentiable Architecture Search with Smooth Activation
- 在架构权重上引入平滑激活函数作为辅助损失，抑制无权重操作的不公平优势。
- 在NAS-Bench-201上达到新SOTA，在分类与超分辨率任务中表现优异。
- 可优化现有高效模型（如信息多蒸馏网络），以更少参数实现更好效果。
可微架构搜索（DARTS）虽高效，但存在鲁棒性差、泛化能力弱及偏差问题。现有方法通过正则化操作权重、路径权重、噪声注入或超网络重构等手段缓解性能坍塌，但难以从根本上解决跳过操作主导的问题。本文提出一种简单有效的方案——平滑激活DARTS（SA-DARTS），通过在架构权重上使用平滑激活函数作为辅助损失，缓解无权重操作的不公平优势，促使架构权重趋于均匀分布，并能从跳过主导的初始状态恢复搜索过程。理论与实证分析表明，SA-DARTS在NAS-Bench-201上的分类与超分辨率任务中均取得新SOTA结果。此外，该方法还能提升现有高效模型（如信息多蒸馏网络）的性能，且参数更少。
原文摘要 · Abstract (English)
Differentiable Architecture Search (DARTS) is an efficient Neural Architecture Search (NAS) method but suffers from robustness, generalization, and discrepancy issues. Many efforts have been made towards the performance collapse issue caused by skip dominance with various regularization techniques towards operation weights, path weights, noise injection, and super-network redesign. It had become questionable at a certain point if there could exist a better and more elegant way to retract the search to its intended goal -- NAS is a selection problem. In this paper, we undertake a simple but effective approach, named Smooth Activation DARTS (SA-DARTS), to overcome skip dominance and discretization discrepancy challenges. By leveraging a smooth activation function on architecture weights as an auxiliary loss, our SA-DARTS mitigates the unfair advantage of weight-free operations, converging to fanned-out architecture weight values, and can recover the search process from skip-dominance initialization. Through theoretical and empirical analysis, we demonstrate that the SA-DARTS can yield new state-of-the-art (SOTA) results on NAS-Bench-201, classification, and super-resolution. Further, we show that SA-DARTS can help improve the performance of SOTA models with fewer parameters, such as Information Multi-distillation Network on the super-resolution task.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。