提出新方法A²M,让神经网络架构搜索更稳定、准确。
Architecture-Aware Minimization (A$^2$M): How to Find Flat Minima in Neural Architecture Search
- 基于架构空间几何特性,设计导向平坦极小值的优化算法
- 在多个数据集上平均提升3.6%~4.6%测试准确率
- 可无缝接入现有NAS框架,适合自动化模型设计研究
神经架构搜索(NAS)已成为设计高效神经网络的关键工具。本文研究了不同可微分NAS方法中常用架构空间的几何特性,聚焦于NAS-Bench-201与DARTS。通过定义邻域和路径上的损失屏障等平坦度度量,揭示了架构空间中类似权重空间损失曲面的局部性与平坦性特征。结果表明,高精度架构聚集于平坦区域,而次优架构则孤立分布,展现了架构搜索空间的详细几何结构。基于此,本文提出架构感知最小化(A²M),一种首次显式将可微分NAS梯度导向架构空间平坦极小值的解析算法框架。A²M在CIFAR-10、CIFAR-100和ImageNet16-120等多个基准数据集上,持续优于当前主流DARTS基线方法。特别地,其在不同可微分NAS方法上平均提升测试准确率:CIFAR-10 (+3.60%)、CIFAR-100 (+4.60%)、ImageNet16-120 (+3.64%),验证了其实际有效性。A²M可轻松集成至现有可微分NAS框架,为自动机器学习研究与应用提供通用工具。代码已开源:https://github.com/AI-Tech-Research-Lab/AsquaredM。
原文摘要 · Abstract (English)
Neural Architecture Search (NAS) has become an essential tool for designing effective and efficient neural networks. In this paper, we investigate the geometric properties of neural architecture spaces commonly used in differentiable NAS methods, specifically NAS-Bench-201 and DARTS. By defining flatness metrics such as neighborhoods and loss barriers along paths in architecture space, we reveal locality and flatness characteristics analogous to the well-known properties of neural network loss landscapes in weight space. In particular, we find that highly accurate architectures cluster together in flat regions, while suboptimal architectures remain isolated, unveiling the detailed geometrical structure of the architecture search landscape. Building on these insights, we propose Architecture-Aware Minimization (A$^2$M), a novel analytically derived algorithmic framework that explicitly biases, for the first time, the gradient of differentiable NAS methods towards flat minima in architecture space. A$^2$M consistently improves generalization over state-of-the-art DARTS-based algorithms on benchmark datasets including CIFAR-10, CIFAR-100, and ImageNet16-120, across both NAS-Bench-201 and DARTS search spaces. Notably, A$^2$M is able to increase the test accuracy, on average across different differentiable NAS methods, by +3.60\% on CIFAR-10, +4.60\% on CIFAR-100, and +3.64\% on ImageNet16-120, demonstrating its superior effectiveness in practice. A$^2$M can be easily integrated into existing differentiable NAS frameworks, offering a versatile tool for future research and applications in automated machine learning. We open-source our code at https://github.com/AI-Tech-Research-Lab/AsquaredM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。