同时优化神经网络结构与权重，用隐空间搜索高效模型。
SWAT-NN: Simultaneous Weights and Architecture Training for Neural Networks in a Latent Space
- 在连续隐空间中联合优化网络结构与权重
- 在合成回归任务上找到稀疏紧凑且性能强的模型
- 适合需要自动设计高效网络的研究者
传统神经网络设计依赖人工试错或先进行神经架构搜索（NAS）再训练权重，前者耗时费力，后者常将架构搜索离散化。本文提出一种新方法，同步优化网络结构与权重。首先训练一个通用多尺度自编码器，将架构和参数信息嵌入连续隐空间，使功能相似的网络在空间中更接近。给定数据集后，随机初始化隐空间一点，通过梯度下降更新该点，获得最优网络，同时优化其结构与权重。优化过程引入稀疏性与紧凑性惩罚，促进高效模型。在合成回归任务上的实验表明，该方法能有效发现稀疏、紧凑且性能优异的神经网络。
原文摘要 · Abstract (English)
Designing neural networks typically relies on manual trial and error or a neural architecture search (NAS) followed by weight training. The former is time-consuming and labor-intensive, while the latter often discretizes architecture search and weight optimization. In this paper, we propose a fundamentally different approach that simultaneously optimizes both the architecture and the weights of a neural network. Our framework first trains a universal multi-scale autoencoder that embeds both architectural and parametric information into a continuous latent space, where functionally similar neural networks are mapped closer together. Given a dataset, we then randomly initialize a point in the embedding space and update it via gradient descent to obtain the optimal neural network, jointly optimizing its structure and weights. The optimization process incorporates sparsity and compactness penalties to promote efficient models. Experiments on synthetic regression tasks demonstrate that our method effectively discovers sparse and compact neural networks with strong performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。