动态调整长文本训练的并行策略，提升效率与平衡性。
DCP: Addressing Input Dynamism In Long-Context Training via Dynamic Context Parallelism
- 按数据块细粒度划分计算与数据，灵活适配不同序列长度。
- 在因果掩码下加速1.19倍至2.45倍，稀疏注意力下提速2.15至3.77倍。
- 适合大规模长文本生成任务，尤其对不规则序列有明显优势。
上下文并行已成为支持长上下文训练的关键技术，这是现代大模型生成AI的发展趋势。然而，现有上下文并行方法依赖静态并行配置，忽略了训练数据的动态特性，即样本间序列长度和注意力模式的差异。这导致通信开销过大且计算负载不均。本文提出DCP，一种动态上下文并行训练框架，引入细粒度的数据与计算块划分，使数据块与计算块可灵活映射到设备上，有效适应不同序列特征，显著降低通信开销，并改善内存与计算平衡。微基准测试显示，在因果掩码下注意力计算加速1.19倍至2.45倍，稀疏注意力下加速2.15至3.77倍；端到端训练速度在因果掩码下提升0.94至1.16倍，稀疏掩码下提升1.00至1.46倍。
原文摘要 · Abstract (English)
Context parallelism has emerged as a key technique to support long-context training, a growing trend in generative AI for modern large models. However, existing context parallel methods rely on static parallelization configurations that overlook the dynamic nature of training data, specifically, the variability in sequence lengths and token relationships (i.e., attention patterns) across samples. As a result, these methods often suffer from unnecessary communication overhead and imbalanced computation. In this paper, we present DCP, a dynamic context parallel training framework that introduces fine-grained blockwise partitioning of both data and computation. By enabling flexible mapping of data and computation blocks to devices, DCP can adapt to varying sequence characteristics, effectively reducing communication and improving memory and computation balance. Micro-benchmarks demonstrate that DCP accelerates attention by 1.19x~2.45x under causal masks and 2.15x~3.77x under sparse attention patterns. Additionally, we observe up to 0.94x~1.16x end-to-end training speed-up for causal masks, and 1.00x~1.46x for sparse masks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。