用神经元交互与图网络提升参数预测，让训练快50%。
Accelerating Training with Neuron Interaction and Nowcasting Networks
- 引入神经元连接建模和图网络改进参数预测
- 在视觉与语言任务中使Adam训练加速最高达50%
- 适合追求高效训练的深度学习研究者
神经网络训练可通过可学习更新规则替代经典自适应优化器（如Adam）来加速。然而，可学习更新规则在训练和使用时可能代价高且不稳定。近期Jang等人（2023）提出一种简化方法：使用权重现在预测网络（WNNs），在多数优化步骤中仍用Adam，仅每隔几步由WNN预测未来参数。本文提出神经元交互与现在预测（NiNo）网络，相较WNNs，利用神经元连接结构与图神经网络更精准地预测参数。针对Transformer等模型中准确建模神经元连接的挑战，我们提出改进方案，克服多项局限，使NiNo在视觉与语言任务中实现最高50%的训练加速。
原文摘要 · Abstract (English)
Neural network training can be accelerated when a learnable update rule is used in lieu of classic adaptive optimizers (e.g. Adam). However, learnable update rules can be costly and unstable to train and use. Recently, Jang et al. (2023) proposed a simpler approach to accelerate training based on weight nowcaster networks (WNNs). In their approach, Adam is used for most of the optimization steps and periodically, only every few steps, a WNN nowcasts (predicts near future) parameters. We improve WNNs by proposing neuron interaction and nowcasting (NiNo) networks. In contrast to WNNs, NiNo leverages neuron connectivity and graph neural networks to more accurately nowcast parameters. We further show that in some networks, such as Transformers, modeling neuron connectivity accurately is challenging. We address this and other limitations, which allows NiNo to accelerate Adam training by up to 50% in vision and language tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。