通过分析初始化时梯度冲突,无需训练就能预测模型性能。
GradAlign for Training-free Model Performance Inference
- 用初始化阶段的样本梯度冲突程度衡量模型潜力
- 在标准基准上优于现有无需训练的架构搜索方法
- 揭示线性区域数作为指标不可靠,适合研究架构设计者
网络架构对深度神经网络性能有重要影响,但最优架构搜索常受限于庞大的搜索空间,导致耗时。近期出现的无训练神经架构搜索(training-free NAS)旨在不经过大量训练即可发现理想架构。该方法利用多种指标进行架构选择,如线性区域数量、每样本损失密度和有限宽度神经正切核(NTK)矩阵的稳定性。尽管当前无训练NAS方法表现良好,但仍存在性能不稳定、机制理解不足的问题。本文提出GradAlign,一种简单有效的无需训练的模型性能推断方法。其核心思想是量化初始化阶段每样本梯度间的冲突程度——冲突越大,收敛越困难,性能越差。在标准NAS基准上评估表明,GradAlign整体性能优于现有方法。此外,我们指出线性区域数量可能不足以作为初始化阶段架构选择的可靠依据。
原文摘要 · Abstract (English)
Architecture plays an important role in deciding the performance of deep neural networks. However, the search for the optimal architecture is often hindered by the vast search space, making it a time-intensive process. Recently, a novel approach known as training-free neural architecture search (NAS) has emerged, aiming to discover the ideal architecture without necessitating extensive training. Training-free NAS leverages various indicators for architecture selection, including metrics such as the count of linear regions, the density of per-sample losses, and the stability of the finite-width Neural Tangent Kernel (NTK) matrix. Despite the competitive empirical performance of current training-free NAS techniques, they suffer from certain limitations, including inconsistent performance and a lack of deep understanding. In this paper, we introduce GradAlign, a simple yet effective method designed for inferring model performance without the need for training. At its core, GradAlign quantifies the extent of conflicts within per-sample gradients during initialization, as substantial conflicts hinder model convergence and ultimately result in worse performance. We evaluate GradAlign against established training-free NAS methods using standard NAS benchmarks, showing a better overall performance. Moreover, we show that the widely adopted metric of linear region count may not suffice as a dependable criterion for selecting network architectures during at initialization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。