动态序列训练时自动切换最优并行策略，解决显存溢出和通信瓶颈。
ParaDySe: A Parallel-Strategy Switching Framework for Dynamic Sequence Lengths in Transformer
- 根据输入序列长度实时选择最佳并行策略，实现动态适配。
- 在长达624K的序列上训练大模型时，避免显存溢出和通信开销过大。
- 适合大规模语言模型训练场景，尤其对长序列数据优化显著。
具有可变长度的动态序列广泛用于基于Transformer的大语言模型（LLM）训练。然而，现有训练框架采用预定义的静态并行策略，导致短序列无法消除通信并行化开销，长序列则容易引发显存溢出。为此，我们提出ParaDySe——一种针对动态序列的自适应并行策略切换框架。ParaDySe通过统一张量布局规范构建模块化并行策略函数库，并结合混合方法建立序列感知的内存与时间成本模型。在成本模型引导下，利用高效启发式算法为每层动态选择最优策略。通过协同集成这些技术，ParaDySe实现了策略的无缝热切换。我们在包含高达624K序列长度的数据集上对代表性LLMs进行测试，结果表明，ParaDySe通过系统整合长序列优化与现有框架，有效缓解了显存溢出（OOM）与通信-并行化冲突（CPC）瓶颈。
原文摘要 · Abstract (English)
Dynamic sequences with varying lengths have been widely used in the training of Transformer-based large language models (LLMs). However, current training frameworks adopt a pre-defined static parallel strategy for these sequences, causing neither communication-parallelization cancellation on short sequences nor out-of-memory on long sequences. To mitigate these issues, we propose ParaDySe, a novel adaptive Parallel strategy switching framework for Dynamic Sequences. ParaDySe enables on-the-fly optimal strategy adoption according to the immediate input sequence. It first implements the modular function libraries for parallel strategies with unified tensor layout specifications, and then builds sequence-aware memory and time cost models with hybrid methods. Guided by cost models, ParaDySe selects optimal layer-wise strategies for dynamic sequences via an efficient heuristic algorithm. By integrating these techniques together, ParaDySe achieves seamless hot-switching of optimal strategies through its well-designed function libraries. We compare ParaDySe with baselines on representative LLMs under datasets with sequence lengths up to 624K. Experimental results indicate that ParaDySe addresses OOM and CPC bottlenecks in LLM training by systematically integrating long-sequence optimizations with existing frameworks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。