提出混合并行方案，提升大模型推荐系统训练效率
Research on Model Parallelism and Data Parallelism Optimization Methods in Large Language Model-Based Recommendation Systems
- 融合张量与流水线并行，动态负载均衡降低通信开销
- 异步模式结合梯度压缩，训练吞吐提升超30%
- 适合大规模推荐系统部署，兼顾性能与可扩展性
随着大语言模型（LLMs）在推荐系统中的广泛应用，其巨大参数量和海量数据带来的计算与通信瓶颈日益突出。本文系统研究了推荐场景下LLM分布式训练的两类优化方法：模型并行与数据并行。在模型并行方面，实现张量并行与流水线并行，并引入自适应负载均衡机制以减少跨设备通信开销；在数据并行方面，对比同步与异步模式，结合梯度压缩与稀疏化技术，构建高效聚合通信框架，显著提升带宽利用率。在真实推荐数据集的模拟服务环境中实验表明，所提出的混合并行方案相比传统单模式并行，训练吞吐提升超过30%，资源利用率提高约20%，同时保持强可扩展性与鲁棒性。最后，讨论了在线部署中不同并行策略的权衡，并展望了异构硬件融合与自动调度技术的未来方向。
原文摘要 · Abstract (English)
With the rapid adoption of large language models (LLMs) in recommendation systems, the computational and communication bottlenecks caused by their massive parameter sizes and large data volumes have become increasingly prominent. This paper systematically investigates two classes of optimization methods-model parallelism and data parallelism-for distributed training of LLMs in recommendation scenarios. For model parallelism, we implement both tensor parallelism and pipeline parallelism, and introduce an adaptive load-balancing mechanism to reduce cross-device communication overhead. For data parallelism, we compare synchronous and asynchronous modes, combining gradient compression and sparsification techniques with an efficient aggregation communication framework to significantly improve bandwidth utilization. Experiments conducted on a real-world recommendation dataset in a simulated service environment demonstrate that our proposed hybrid parallelism scheme increases training throughput by over 30% and improves resource utilization by approximately 20% compared to traditional single-mode parallelism, while maintaining strong scalability and robustness. Finally, we discuss trade-offs among different parallel strategies in online deployment and outline future directions involving heterogeneous hardware integration and automated scheduling technologies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。