提出GNN超参数迁移方法,让小模型调好的参数直接用在大模型上。
Hyperparameter Transfer in Graph Neural Networks

- 基于SGD、Adam、AdamW设计可迁移的超参数配置方案
- 在图神经网络中实现宽度与深度增加时性能稳定提升
- 适合需要高效扩展GNN模型的研究者和工程师
深度学习模型的性能高度依赖于学习率、初始化尺度和权重衰减等超参数设置。超参数迁移旨在使模型规模变化时仍保持近似最优的超参数配置,从而通过优化小型廉价模型来代理大模型的训练。尽管该原理在语言和视觉任务的密集神经网络中已得到充分研究,但在图神经网络(GNN)中仍相对未被深入探索。本文为使用SGD、Adam和AdamW训练的GNN开发并验证了一种超参数迁移参数化方法。通过理论缩放分析与受控实验,证明该方法能实现稳定的特征更新、学习率迁移,并在模型宽度和深度增加时提升性能。对于SGD,识别出与图结构相关的首层修正因子,其应用可加速稀疏词袋输入图的早期训练;对于Adam,分析了不同消息传递归一化对早/晚训练阶段迁移行为的影响,揭示了消息传递归一化的关键作用并建议引入相应超参数;对于AdamW,采用一种联合迁移权重衰减与学习率的参数化策略。这些成果为多种学习任务和训练场景下的GNN规模化提供了实用指导。
原文摘要 · Abstract (English)
The performance of deep learning models crucially depends on the settings of hyperparameters like learning rate, initialization scale, and weight decay. Hyperparameter transfer aims to make near-optimal hyperparameter settings consistent across model scale, so that large models can be optimized by proxy tuning their smaller, cheaper-to-optimize counterparts. While transfer principles are well-studied in the context of dense neural networks in language and vision tasks, they remain comparatively under-explored for graph neural networks (GNNs). We develop and validate a transfer parameterization for GNNs trained with SGD, Adam, and AdamW. Through theoretical scaling analyses and controlled experiments, we show that the proposed parameterization yields stable feature updates, learning rate transfer, and improved performance as width and depth increase. For SGD, we identify graph-dependent first-layer correction factors and show that their use can accelerate early training in graphs with sparse bag-of-words inputs. For Adam, we explore how different message passing normalizations affect early- and late-training transfer behavior, illustrating the importance of message passing normalization and advocating for an associated hyperparameter. For AdamW, we adapt a parameterization that allows for the joint transfer of weight decay and learning rate. Together, these results provide a practical recipe for scaling GNNs across a variety of learning tasks and training scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。