通过梯度方向分析,实现高效少样本图神经网络架构搜索。
Towards Efficient Few-shot Graph Neural Architecture Search via Partitioning Gradient Contribution
- 基于梯度方向相似性划分超网模块,缓解权重耦合问题。
- 所提方法在分区质量与计算效率上均达当前最优水平。
- 适用于探索MPNN与图变压器的混合架构,适合研究者参考。
为解决权重耦合问题,现有少样本神经架构搜索(NAS)方法将超网划分为多个子超网,但常面临计算低效和分区方案不佳的问题。本文从新视角分析该问题,发现相邻层模块间梯度方向冲突是主要原因。为此,提出梯度贡献(GC)方法,在超网反向传播中分解向量-雅可比积,高效计算模块间梯度方向的余弦相似度。具有冲突梯度方向的模块被分配至不同子超网,相似者则归为一组。为克服现有图神经网络架构搜索仅限于单一类型(如消息传递网络或图变压器)的局限,提出统一图神经网络架构搜索(UGAS)框架,探索MPNN与GT的最优组合。实验表明,GC在超网分区质量和时间效率上均达当前最优;由UGAS+GC搜索出的架构优于人工设计及已有NAS方法所得模型。消融实验进一步验证了各方法的有效性。
原文摘要 · Abstract (English)
To address the weight coupling problem, certain studies introduced few-shot Neural Architecture Search (NAS) methods, which partition the supernet into multiple sub-supernets. However, these methods often suffer from computational inefficiency and tend to provide suboptimal partitioning schemes. To address this problem more effectively, we analyze the weight coupling problem from a novel perspective, which primarily stems from distinct modules in succeeding layers imposing conflicting gradient directions on the preceding layer modules. Based on this perspective, we propose the Gradient Contribution (GC) method that efficiently computes the cosine similarity of gradient directions among modules by decomposing the Vector-Jacobian Product during supernet backpropagation. Subsequently, the modules with conflicting gradient directions are allocated to distinct sub-supernets while similar ones are grouped together. To assess the advantages of GC and address the limitations of existing Graph Neural Architecture Search methods, which are limited to searching a single type of Graph Neural Networks (Message Passing Neural Networks (MPNNs) or Graph Transformers (GTs)), we propose the Unified Graph Neural Architecture Search (UGAS) framework, which explores optimal combinations of MPNNs and GTs. The experimental results demonstrate that GC achieves state-of-the-art (SOTA) performance in supernet partitioning quality and time efficiency. In addition, the architectures searched by UGAS+GC outperform both the manually designed GNNs and those obtained by existing NAS methods. Finally, ablation studies further demonstrate the effectiveness of all proposed methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。