提出新模型，通过关注节点兄弟关系提升神经网络结构预测精度
NN-Former: Rethinking Graph Structure in Neural Architecture Representation
- 引入兄弟节点感知的token混合机制，改进图结构建模
- 在准确率与延迟预测上均超越现有方法，尤其在深层网络表现更优
- 适合需要高效神经网络设计的研究者和工程师
深度学习广泛应用推动了高效网络设计与部署的需求，神经预测器在估算精度与延迟等属性方面日益重要。近年来，图神经网络（GNN）与Transformer在神经架构表示中表现出色，但各有缺陷：GNN难以捕捉复杂特征，而Transformer在架构深度增加时泛化能力下降。本文重新思考神经架构拓扑结构，发现兄弟节点在以往研究中被忽视但至关重要。为此，提出一种融合GNN与Transformer优势的新预测模型，引入考虑兄弟关系的新型token混合器，以及名为双向图同构前馈网络的新通道混合模块。该方法在准确率与延迟预测任务中持续表现优异，为有向无环图（DAG）拓扑学习提供了新思路。代码已开源：https://github.com/XuRuihan/NNFormer。
原文摘要 · Abstract (English)
The growing use of deep learning necessitates efficient network design and deployment, making neural predictors vital for estimating attributes such as accuracy and latency. Recently, Graph Neural Networks (GNNs) and transformers have shown promising performance in representing neural architectures. However, each of both methods has its disadvantages. GNNs lack the capabilities to represent complicated features, while transformers face poor generalization when the depth of architecture grows. To mitigate the above issues, we rethink neural architecture topology and show that sibling nodes are pivotal while overlooked in previous research. We thus propose a novel predictor leveraging the strengths of GNNs and transformers to learn the enhanced topology. We introduce a novel token mixer that considers siblings, and a new channel mixer named bidirectional graph isomorphism feed-forward network. Our approach consistently achieves promising performance in both accuracy and latency prediction, providing valuable insights for learning Directed Acyclic Graph (DAG) topology. The code is available at https://github.com/XuRuihan/NNFormer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。