针对大模型推荐系统在昇腾芯片上的训练瓶颈,提出高效加速方案。
TurboGR: An Accelerated Training System for Large-Scale Generative Recommendation

- 设计适配昇腾的稀疏操作融合与动态负载均衡,消除填充开销。
- 实现94%芯片利用率和0.97的近线性扩展能力,训练效率显著提升。
- 适合大规模生成式推荐系统研发者,尤其关注昇腾硬件优化的团队。
生成式推荐(GR)作为统一的Transformer架构范式,展现出随模型规模和数据量增长而系统提升的缩放规律。然而,在昇腾NPU上部署大尺度GR面临根本性的系统挑战:稀疏操作缺乏高性能实现,且不规则稀疏原语与NPU密集计算优化架构不匹配。本文提出 model,一个面向昇腾的生成式推荐训练系统,通过三项核心创新系统性解决这些瓶颈:(i) 昇腾适配的稀疏加速,包括消除填充冗余的操作融合,以及将设备间负载不平衡从47%降至2.4%的动态均衡;(ii) 分布式通信优化,包含分层稀疏并行、具备收敛保证的半异步训练及细粒度流水线调度,维持94% NPU利用率;(iii) 通过异步卸载、感知稀疏性的FP16量化和批内对数共享,扩大有效负样本空间而无需额外嵌入查找。在KuaiRand-27K数据集上评估,model 支持高达0.2B参数的训练,达到54.71%的MFU,且具有接近线性的可扩展性(0.97)。
原文摘要 · Abstract (English)
Generative recommendation (GR) has emerged as a promising paradigm that replaces fragmented, scenario-specific architectures with unified Transformer-based models, exhibiting scaling-law behavior where recommendation quality improves systematically with increased model capacity and training data. However, deploying GR at scale on Ascend NPUs faces fundamental system-level challenges. These challenges are further exacerbated on Ascend NPUs due to the absence of high-performance implementations for jagged operators and the architectural mismatch between irregular sparse primitives and NPU's dense-computation-optimized design. In this paper, we present \model, an Ascend-affinity training system for generative recommendation that systematically addresses these bottlenecks through three core innovations: (i) Ascend-affinity jagged acceleration, including fusion operators that eliminate padding redundancy and dynamic load balancing that reduces inter-device imbalance from 47\% to 2.4\%; (ii) distributed communication optimization, comprising hierarchical sparse parallelism, semi-asynchronous training with proven convergence guarantees, and fine-grained pipeline orchestration that sustains 94\% NPU utilization; and (iii) negative sampling optimization via asynchronous offloading, jaggedness-aware FP16 quantization, and intra-batch logit sharing that expand the effective negative space without additional embedding lookups. Evaluated on the KuaiRand-27K dataset, \model supports training at up to 0.2B parameters and achieves 54.71\% MFU with near-linear scalability (0.97).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。