提出DHO₂加速分布式训练，用模型并行降低内存压力。
DHO$_2$: Accelerating Distributed Hybrid Order Optimization via Model Parallelism and ADMM
- 用模型并行分摊曲率信息计算，减少单设备内存占用。
- 相比传统优化器，训练时间提速1.4到2.1倍。
- 适合资源有限但需快速训练大模型的场景。
将深度神经网络（DNN）训练扩展到更多设备可缩短求解时间，但对资源受限用户不现实。FOSI作为混合阶优化器，通过结合梯度与曲率信息更新模型，收敛速度优于传统优化器，为资源受限环境下的训练加速带来新可能。本文探索其分布式设计DHO₂，包括曲率信息的分布式计算及部分曲率信息下的模型更新，以在低内存负担下加速训练。为进一步缩短训练时间，设计了一种新策略，将曲率信息计算与模型更新分别在不同设备上并行执行。实验表明，该分布式设计使每台设备的内存负担随设备数增加近似线性下降；同时，相较于基于一阶和二阶优化器的其他分布式方案，总训练时间提速1.4至2.1倍。
原文摘要 · Abstract (English)
Scaling deep neural network (DNN) training to more devices can reduce time-to-solution. However, it is impractical for users with limited computing resources. FOSI, as a hybrid order optimizer, converges faster than conventional optimizers by taking advantage of both gradient information and curvature information when updating the DNN model. Therefore, it provides a new chance for accelerating DNN training in the resource-constrained setting. In this paper, we explore its distributed design, namely DHO$_2$, including distributed calculation of curvature information and model update with partial curvature information to accelerate DNN training with a low memory burden. To further reduce the training time, we design a novel strategy to parallelize the calculation of curvature information and the model update on different devices. Experimentally, our distributed design can achieve an approximate linear reduction of memory burden on each device with the increase of the device number. Meanwhile, it achieves $1.4\times\sim2.1\times$ speedup in the total training time compared with other distributed designs based on conventional first- and second-order optimizers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。