用预训练任务让编码器学会捕捉网络信息流动，提升性能同时加快速度。
Learning to Flow from Generative Pretext Tasks for Neural Architecture Encoding
- 通过重建信息流代理表示来预训练编码器，无需复杂结构。
- 在Precision-1%上性能提升最高达106%。
- 适合需要高效准确架构评估的研究者。
深度学习模型在特定任务和数据集上的表现高度依赖其神经网络架构，促使研究者致力于快速准确地识别适配的架构。为此，通常使用机器学习模型（尤其是神经架构编码器）预测架构性能。现有先进编码器通过特殊模型结构捕捉信息流，反映前向传播与反向传播中信息的传递方式。然而，这些基于流的编码器因结构复杂，处理速度远慢于简单编码器，带来显著实践挑战。为此，我们提出FGP——一种新型神经架构编码的预训练方法，使编码器在无需特殊结构的情况下也能捕捉信息流。FGP通过训练编码器重构我们提出的神经架构信息流代理表示。实验表明，相较于仅使用监督学习训练的同一编码器，FGP在Precision-1%上性能提升高达106%。
原文摘要 · Abstract (English)
The performance of a deep learning model on a specific task and dataset depends heavily on its neural architecture, motivating considerable efforts to rapidly and accurately identify architectures suited to the target task and dataset. To achieve this, researchers use machine learning models-typically neural architecture encoders-to predict the performance of a neural architecture. Many state-of-the-art encoders aim to capture information flow within a neural architecture, which reflects how information moves through the forward pass and backpropagation, via a specialized model structure. However, due to their complicated structures, these flow-based encoders are significantly slower to process neural architectures compared to simpler encoders, presenting a notable practical challenge. To address this, we propose FGP, a novel pre-training method for neural architecture encoding that trains an encoder to capture the information flow without requiring specialized model structures. FGP trains an encoder to reconstruct a flow surrogate, our proposed representation of the neural architecture's information flow. Our experiments show that FGP boosts encoder performance by up to 106% in Precision-1%, compared to the same encoder trained solely with supervised learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。