arXiv:2508.12977cs.CV2025-08中稿 · Transactions on Ma…被引 2

无需标签数据,用奇异值分解和曲率预测网络性能。

Dextr: Zero-Shot Neural Architecture Search with Singular Value Decomposition and Extrinsic Curvature

  • 结合特征条件数与输出曲率,设计无标签数据的零成本代理。
  • 在DARTS和AutoFormer搜索空间中相关性超越现有方法。
  • 适合追求高效、无需标注数据的神经网络架构搜索场景。

零样本神经网络架构搜索(NAS)通常通过初始状态下的网络或梯度特性来优化搜索过程,但现有代理大多依赖标签数据,而真实场景中常不可得。此外,多数方法仅关注收敛性或泛化性,或仅关注表达能力。本文首次揭示通道共线性对收敛性和泛化性的影响,并提出一种统一融合收敛性、泛化性和表达性的零成本代理,无需标签数据即可计算。该代理基于神经网络层特征的奇异值分解(SVD)及网络输出的外在曲率,其形式为特征条件数倒数之和与输出曲率对数的调和平均的简化版。仅需单个无标签样本,即可准确预测测试性能。实验涵盖六个任务:CNN搜索空间(DARTS)与Transformer搜索空间(AutoFormer),在NAS-Bench-101、NAS-Bench-201、TransNAS-Bench-101-micro等多个相关性基准上表现优异,且计算效率高。代码已开源。

原文摘要 · Abstract (English)

Zero-shot Neural Architecture Search (NAS) typically optimises the architecture search process by exploiting the network or gradient properties at initialisation through zero-cost proxies. The existing proxies often rely on labelled data, which is usually unavailable in real-world settings. Furthermore, the majority of the current methods focus either on optimising the convergence and generalisation attributes or solely on the expressivity of the network architectures. To address both limitations, we first demonstrate how channel collinearity affects the convergence and generalisation properties of a neural network. Then, by incorporating the convergence, generalisation and expressivity in one approach, we propose a zero-cost proxy that omits the requirement of labelled data for its computation. In particular, we leverage the Singular Value Decomposition (SVD) of the neural network layer features and the extrinsic curvature of the network output to design our proxy. %As a result, the proposed proxy is formulated as the simplified harmonic mean of the logarithms of two key components: the sum of the inverse of the feature condition number and the extrinsic curvature of the network output. Our approach enables accurate prediction of network performance on test data using only a single label-free data sample. Our extensive evaluation includes a total of six experiments, including the Convolutional Neural Network (CNN) search space, i.e. DARTS and the Transformer search space, i.e. AutoFormer. The proposed proxy demonstrates a superior performance on multiple correlation benchmarks, including NAS-Bench-101, NAS-Bench-201, and TransNAS-Bench-101-micro; as well as on the NAS task within the DARTS and the AutoFormer search space, all while being notably efficient. The code is available at https://github.com/rohanasthana/Dextr.

零样本NAS奇异值分解架构搜索无标签评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。