让数据自适应调度，实现长序列训练的高效与易用兼顾
Training Variable Long Sequences with Data-Centric Parallel
- 根据每批数据长度动态调整并行规模和梯度累积
- 在32张H200 GPU上实现最高2.88倍加速
- 仅需10行代码即可适配任意模型
在可变长度序列上训练深度学习模型面临显著计算挑战。现有方法在效率与易用性之间陷入两难：简单方法采用静态配置导致负载不均、效率低下；复杂方法则引入大量代码改动与实现复杂性。为此，我们提出数据驱动的并行框架（Data-Centric Parallel, DCP），其核心思想是让数据本身决定运行时行为。通过动态调整并行规模、梯度累积和梯度重计算等运行时参数，以适应每批次序列长度的变化。实证结果表明，该方法在32张H200 GPU上实现了最高2.88倍的加速。该方案具备强泛化能力，仅需约10行代码即可集成至任意模型中。我们预计这一简单而有效的方法将成为可变长序列分布式训练的稳健基线，并推动后续技术发展。
原文摘要 · Abstract (English)
Training deep learning models on variable long sequences poses significant computational challenges. Existing methods force a difficult trade-off between efficiency and ease-of-use. Simple approaches use static configurations that cause workload imbalance low efficiency, while complex methods introduces significant complexity and code change for new models. To break this trade-off, we introduce Data-Centric Parallel (DCP). Its core principle is to let the data itself drive the runtime. It achieves this by dynamically adjusting direct runtime settings (e.g., parallel size, gradient accumulation, recomputation) based on each batch's sequence length. Empirical results demonstrate that our method achieves up to a 2.88$\times$ speedup on 32 H200 GPUs. Designed for generalization, it can be integrated into any model with 10 lines of code. We anticipate this simple yet effective approach will serve as a robust baseline and facilitate future advancements in distributed training for variable long sequences.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。