用自然语言描述统一表示网络,实现任意架构的即时性能预测。
ONNX-Net: Towards Universal Representations and Instant Performance Prediction for Neural Architectures
- 用ONNX文件统一表示网络,通过文本描述建模
- 仅需少量预训练样本即在多类搜索空间中实现零样本预测
- 适合需要快速评估新架构的研究者和开发者
神经架构搜索(NAS)自动化设计高性能网络,但受限于昂贵的性能评估。现有方法大多依赖特定细胞结构的搜索空间和图编码,难以拓展到更复杂的搜索空间。本文提出ONNX-Bench,一个基于ONNX格式的基准数据集,包含超过60万条{架构, 准确率}对,涵盖所有开源NAS-bench中的网络。在此基础上构建通用网络表示ONNX-Net,通过自然语言描述作为输入,实现任意架构的性能预测。该文本编码可支持任意层类型、参数和异构拓扑,使单一代理模型能跨多种架构泛化,无需针对特定搜索空间定制。实验表明,仅需少量预训练样本即可在不同搜索空间中实现强零样本性能,首次实现对任意神经网络架构的即时评估。
原文摘要 · Abstract (English)
Neural architecture search (NAS) automates the design process of high-performing architectures, but remains bottlenecked by expensive performance evaluation. Most existing studies that achieve faster evaluation are mostly tied to cell-based search spaces and graph encodings tailored to those individual search spaces, limiting their flexibility and scalability when applied to more expressive search spaces. In this work, we aim to close the gap of individual search space restrictions and search space dependent network representations. We present ONNX-Bench, a benchmark consisting of a collection of neural networks in a unified format based on ONNX files. ONNX-Bench includes all open-source NAS-bench-based neural networks, resulting in a total size of more than 600k {architecture, accuracy} pairs. This benchmark allows creating a shared neural network representation, ONNX-Net, able to represent any neural architecture using natural language descriptions acting as an input to a performance predictor. This text-based encoding can accommodate arbitrary layer types, operation parameters, and heterogeneous topologies, enabling a single surrogate to generalise across all neural architectures rather than being confined to cell-based search spaces. Experiments show strong zero-shot performance across disparate search spaces using only a small amount of pretraining samples, enabling the unprecedented ability to evaluate any neural network architecture instantly.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。