提出自适应动态并行策略，高效训练长文本大模型。
Efficient Scaling of LLM Training with Flexible Context Parallelism
- 根据数据长度动态调整通信组和并行度，避免资源浪费。
- 在极端不均衡数据下提升2.24倍训练速度，平均提速1.46倍。
- 适合大规模长文本训练，尤其适用于数据长度差异大的场景。
扩展长上下文能力对大语言模型（LLM）至关重要。然而，真实数据中序列长度差异显著。现有训练框架采用静态并行策略，在数据异构时存在严重负载不均、冗余通信和硬件利用率低下问题。本文提出灵活上下文并行（FCP），可在训练过程中自适应重构通信组与上下文并行度。支持非2的幂次并行度，并设计多项式时间算法，每批次仅需毫秒级开销生成近优并行策略。实验表明，FCP在大规模集群上保持高硬件效率，相比Megatron-LM和DeepSpeed，在LLM与多模态大模型训练中平均吞吐提升1.46倍，极端不均衡批次下最高达2.24倍加速，且实现接近线性的扩展效率。
原文摘要 · Abstract (English)
Scaling long-context capabilities is crucial for Large Language Models (LLMs). However, real-world data contain a large number of sequences with heterogeneous lengths. Existing training libraries for LLMs rely on static parallelism strategies, which suffer from severe load imbalance, redundant communication, and suboptimal hardware utilization under data heterogeneity. In this work, we propose Flexible Context Parallelism (FCP), an efficient parallelism strategy that adaptively reconfigures communication groups and context parallelism degrees during LLM training. We generalize more flexible non-power-of-two parallelism degrees and develop a polynomial-time algorithm to generate near-optimal parallelism strategies with only millisecond-level overhead per training batch. FCP is able to maintain high hardware efficiency even under extreme data heterogeneity. Experimental results demonstrate that FCP significantly outperforms Megatron-LM and DeepSpeed in both LLM and MLLM training, achieving up to 1.46x speedup in average throughput while maintaining near-linear scaling efficiency across large-scale clusters. For extremely unbalanced batches, FCP even achieves 2.24x speedup.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。