用优化代理将神经网络搜索转为端到端优化，提速且效果更好。
OptiProxy-NAS: Optimization Proxy based End-to-End Neural Architecture Search
- 构建连续可微的代理表示，把离散搜索空间变为平滑优化问题。
- 在12个任务中平均节省80%时间，准确率超越主流方法。
- 适合追求高效搜索的工程师和资源受限场景的研究者。
神经架构搜索（NAS）是一个计算成本高、离散且结构复杂的优化难题。现有方法多依赖预测器或可微超网，而本文提出一种基于优化代理的端到端搜索框架——OptiProxy-NAS。通过引入代理表示，将原始离散搜索空间重构为连续、可微且平滑的空间，使任意可微优化方法均可用于梯度驱动的架构参数搜索。我们在四个不同搜索空间的12个任务上进行实验，涵盖计算机视觉、自然语言处理及资源受限场景，全面验证了该方法在搜索效率与性能上的优越性。此外，在低保真场景下的实验进一步证明其灵活性。
原文摘要 · Abstract (English)
Neural architecture search (NAS) is a hard computationally expensive optimization problem with a discrete, vast, and spiky search space. One of the key research efforts dedicated to this space focuses on accelerating NAS via certain proxy evaluations of neural architectures. Different from the prevalent predictor-based methods using surrogate models and differentiable architecture search via supernetworks, we propose an optimization proxy to streamline the NAS as an end-to-end optimization framework, named OptiProxy-NAS. In particular, using a proxy representation, the NAS space is reformulated to be continuous, differentiable, and smooth. Thereby, any differentiable optimization method can be applied to the gradient-based search of the relaxed architecture parameters. Our comprehensive experiments on $12$ NAS tasks of $4$ search spaces across three different domains including computer vision, natural language processing, and resource-constrained NAS fully demonstrate the superior search results and efficiency. Further experiments on low-fidelity scenarios verify the flexibility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。