提出新零样本评估方法SWAP-Score，跨架构与任务通用且精度高。
Zero-Shot Neural Network Evaluation with Sample-Wise Activation Patterns
- 基于样本激活模式衡量网络表达能力，无需训练即可评估性能。
- 在图像与文本任务中相关性高达0.93（CNN）和0.71（Transformer）。
- 适用于预训练阶段与NAS，显著降低计算成本，适合高效模型筛选。
零样本代理（zero-shot proxies），又称训练自由度量，广泛用于神经网络评估以降低计算开销，尤其在神经架构搜索（NAS）中。现有方法存在与真实性能相关性弱、跨网络或下游任务泛化能力差等问题，多数仅适用于卷积神经网络（CNN）或Transformer，不具通用性。为此，本文提出样本级激活模式（SWAP）及其衍生指标SWAP-Score，一种新颖且高效的零样本度量。该方法在小批量样本上衡量网络表达能力，表现出对真实性能的强预测性。在计算机视觉与自然语言处理任务中，对CNN和Transformer均表现优异：在DARTS CNN于CIFAR-10上的验证准确率中，SWAP-Score与真实性能的斯皮尔曼相关系数达0.93；在FlexiBERT Transformer于GLUE任务上为0.71。此外，该方法不依赖标签，可用于语言模型预训练阶段估算下游任务表现。在NAS中，基于SWAP的SWAP-NAS仅需约6分钟（CIFAR-10）和9分钟（ImageNet）GPU时间即达到可比性能。代码已开源。
原文摘要 · Abstract (English)
Zero-shot proxies, also known as training-free metrics, are widely adopted to reduce the computational overhead in neural network evaluation for scenarios such as Neural Architecture Search (NAS), as they do not require any training. Existing zero-shot metrics have several limitations, including weak correlation with the true performance and poor generalisation across different networks or downstream tasks. For example, most of these metrics apply only to either convolutional neural networks (CNNs) or Transformers, but not both. To address these limitations, we propose Sample-Wise Activation Patterns (SWAP), and its derivative, SWAP-Score, a novel and highly effective zero-shot metric. SWAP-Score is broadly applicable across both architecture families and task domains, demonstrating strong predictive performance in the majority of tasks. This metric measures the expressivity of neural networks over a mini-batch of samples, showing a high correlation with the neural networks' ground-truth performance. For both CNNs and Transformers, the SWAP-Score outperforms existing zero-shot metrics across computer vision and natural language processing tasks. For instance, Spearman's correlation coefficient between the SWAP-Score and CIFAR-10 validation accuracy for DARTS CNNs is 0.93, and 0.71 for FlexiBERT Transformers on GLUE tasks. Moreover, SWAP-Score is label-independent, hence can be applied at the pre-training stage of language models to estimate their performance for downstream tasks. When applied to NAS, SWAP-empowered NAS, SWAP-NAS can achieve competitive performance using only approximately 6 and 9 minutes of GPU time, on CIFAR-10 and ImageNet respectively. Our code is available at: https://github.com/pym1024/SWAP_Universal
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。