通过嵌套流水线实现千卡规模推荐系统高效训练
NestPipe: Large-Scale Recommendation Training on 1,500+ Accelerators via Nested Pipelining
- 采用双缓冲与冻结窗口流水线，分层优化数据查找和通信延迟
- 在1536个计算单元上实现3.06倍加速和94.07%扩展效率
- 适合大规模推荐系统训练，兼顾速度与训练一致性
现代推荐模型参数量已达万亿级。随着集群规模扩展至千级，分布式训练瓶颈从计算与内存转向数据移动，尤其是嵌入向量的查找与通信延迟。现有方案或仅优化单一瓶颈，或通过牺牲训练一致性提升吞吐量。本文提出NestPipe，一种大规模去中心化嵌入训练框架，同时解决两类瓶颈并保持同步训练语义。NestPipe通过嵌套流水线挖掘两级稀疏并行机会：在批间层面，双缓冲流水线（DBP）利用双缓冲同步构建无过时五阶段流水线，缓解查找瓶颈且无嵌入过时；在批内层面，发现嵌入冻结现象，提出冻结窗口流水线（FWP），通过协调流调度与键中心样本聚类，使All2All通信与密集计算重叠。在含1536个工作者的生产级GPU与NPU集群上的实验表明，NestPipe实现最高3.06倍加速和94.07%扩展效率。
原文摘要 · Abstract (English)
Modern recommendation models have increased to trillions of parameters. As cluster scales expand to O(1k), distributed training bottlenecks shift from computation and memory to data movement, especially lookup and communication latency associated with embeddings. Existing solutions either optimize only one bottleneck or improve throughput by sacrificing training consistency. This paper presents NestPipe, a large-scale decentralized embedding training framework that tackles both bottlenecks while preserving synchronous training semantics. NestPipe exploits two hierarchical sparse parallelism opportunities through nested pipelining. At the inter-batch level, Dual-Buffer Pipelining (DBP) constructs a staleness-free five-stage pipeline through dual-buffer synchronization, mitigating lookup bottlenecks without embedding staleness. At the intra-batch level, we identify the embedding freezing phenomenon, which inspires Frozen-Window Pipelining (FWP) to overlap All2All communication with dense computation via coordinated stream scheduling and key-centric sample clustering. Experiments on production GPU and NPU clusters with 1,536 workers demonstrate that NestPipe achieves up to 3.06x speedup and 94.07% scaling efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。