FreeScale降低序列推荐模型训练中的计算浪费，提升大规模训练效率。
FreeScale: Distributed Training for Sequence Recommendation Models with Minimal Scaling Cost
- 通过负载均衡输入样本缓解慢节点问题
- 通信与计算重叠，减少阻塞通信时间
- 采用SM-Free技术避免显卡资源竞争，适合工业级推荐系统
现代工业级深度学习推荐模型通常通过分析用户的历史交互序列来提取偏好，并基于这些兴趣生成预测。由于数据特征的固有异构性，大规模训练中常出现计算资源严重闲置，主要源于慢节点导致的计算气泡和阻塞通信。本文提出FreeScale，通过三方面优化：(1) 精细负载均衡输入样本以缓解慢节点问题；(2) 将高优先级嵌入通信与计算重叠，最小化阻塞通信；(3) 在计算与通信重叠时通过SM-Free技术解决GPU资源竞争。实证评估显示，将FreeScale应用于真实工作负载（256张H100 GPU），计算气泡最多可减少90.3%。
原文摘要 · Abstract (English)
Modern industrial Deep Learning Recommendation Models typically extract user preferences through the analysis of sequential interaction histories, subsequently generating predictions based on these derived interests. The inherent heterogeneity in data characteristics frequently result in substantial under-utilization of computational resources during large-scale training, primarily due to computational bubbles caused by severe stragglers and slow blocking communications. This paper introduces FreeScale, a solution designed to (1) mitigate the straggler problem through meticulously load balanced input samples (2) minimize the blocking communication by overlapping prioritized embedding communications with computations (3) resolve the GPU resource competition during computation and communication overlapping by communicating through SM-Free techniques. Empirical evaluation demonstrates that FreeScale achieves up to 90.3% reduction in computational bubbles when applied to real-world workloads running on 256 H100 GPUs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。