arXiv:2503.22063cs.LG2025-03

用离散向量量化方法学习神经网络结构,提升生成有效架构的效率。

Arch-VQ: Discrete Architecture Representation Learning with Autoregressive Priors

  • 采用VQ-VAE与自回归Transformer构建离散架构表示空间。
  • 在三个基准上生成有效且唯一的架构率分别提升22%~135%。
  • 适合需要高效生成可靠神经网络结构的研究者使用。

现有神经架构表示学习方法多依赖连续表示,通常通过变分自编码器(VAE)将离散架构映射到连续高斯分布,但采样常产生大量无效或重复结构,可能源于离散架构空间与连续空间的不匹配。本文从离散视角重新思考架构表示学习,提出Arch-VQ框架:利用向量量化变分自编码器(VQ-VAE)学习离散潜在空间,并以自回归Transformer建模潜变量先验。该设计使表示更契合真实搜索空间,同时解耦表示学习与先验建模。在NASBench-101、NASBench-201和DARTS搜索空间上,Arch-VQ相比当前最优基线,有效且唯一生成架构的比例分别提升22%、26%和135%。进一步实验表明,离散嵌入的自回归建模可提升下游神经预测器性能,验证了该离散范式实际价值。

原文摘要 · Abstract (English)

Existing neural architecture representation learning methods focus on continuous representation learning, typically using Variational Autoencoders (VAEs) to map discrete architectures onto a continuous Gaussian distribution. However, sampling from these spaces often leads to a high percentage of invalid or duplicate neural architectures, likely due to the unnatural mapping of inherently discrete architectural space onto a continuous space. In this work, we revisit architecture representation learning from a fundamentally discrete perspective. We propose Arch-VQ, a framework that learns a discrete latent space of neural architectures using a Vector-Quantized Variational Autoencoder (VQ-VAE), and models the latent prior with an autoregressive transformer. This formulation yields discrete architecture representations that are better aligned with the underlying search space while decoupling representation learning from prior modeling. Across NASBench-101, NASBench-201, and DARTS search spaces, Arch-VQ improves the quality of generated architectures, increasing the rate of valid and unique generations by 22%, 26%, and 135%, respectively, over state-of-the-art baselines. We further show that modeling discrete embeddings autoregressively enhances downstream neural predictor performance, establishing the practical utility of this discrete formulation.

架构搜索离散表示VQ-VAE自回归模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。