解决异构集群训练慢、不准问题，动态调参提升效率与精度。
OmniLearn: A Framework for Distributed Deep Learning over Heterogeneous Clusters
- 基于比例控制思想动态调整各节点小批量大小，平衡计算负载。
- 在异构环境下训练时间缩短14%至85%，异步训练准确率最高提升6.9%。
- 适合边缘、云、超算混合环境的分布式深度学习部署场景。
深度学习系统通常针对资源同质的集群进行优化，但边缘、云和高性能计算（HPC）环境中普遍存在异构性。在异构资源上使用随机梯度下降训练神经网络时，因慢节点（stragglers）和过时更新导致性能下降。本文提出一种自适应批处理扩展框架OmniLearn，通过比例控制器思想实现异构服务器间的计算负载均衡，可在资源动态变化下运行。该方法在运行时动态调整工作节点的迷你批次大小，使训练时间减少14%-85%。同时，研究了异步训练场景，所提技术可将准确率提升最高达6.9%。
原文摘要 · Abstract (English)
Deep learning systems are optimized for clusters with homogeneous resources. However, heterogeneity is prevalent in computing infrastructure across edge, cloud and HPC. When training neural networks using stochastic gradient descent techniques on heterogeneous resources, performance degrades due to stragglers and stale updates. In this work, we develop an adaptive batch-scaling framework called OmniLearn to mitigate the effects of heterogeneity in distributed training. Our approach is inspired by proportional controllers to balance computation across heterogeneous servers, and works under varying resource availability. By dynamically adjusting worker mini-batches at runtime, OmniLearn reduces training time by 14-85%. We also investigate asynchronous training, where our techniques improve accuracy by up to 6.9%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。