提出分层均衡打包方法，显著加速长文本大模型训练。
Hierarchical Balance Packing: Towards Efficient Supervised Fine-tuning for Long-Context LLM
- 设计多层级数据打包组，按最优长度分配样本
- 在236B模型上训练提速2.4倍，性能保持优秀
- 适合大规模长上下文模型高效微调的场景
训练长上下文大语言模型面临挑战，混合使用长、短上下文数据常导致计算负载不均。现有方法主要依赖数据打包缓解此问题，但未考虑注意力计算不均衡和通信开销浪费。本文提出分层均衡打包（HBP），设计新型批处理构建方法与训练方案。HBP构建多层级数据打包组，每组采用不同最优打包长度，并将样本分配至最适配组，配置相应的流水并行度与梯度检查点策略。为有效利用多层级组，设计了专为HBP定制的动态训练管道，包含课程学习、自适应流水并行与稳定损失机制。大量实验表明，该方法在多个数据集和开源模型上显著缩短训练时间，且保持优异性能。在最大规模的DeepSeek-V2（236B）MoE模型上，训练速度提升2.4倍，性能相当。代码将公开于https://github.com/ModelTC/HBP。
原文摘要 · Abstract (English)
Training Long-Context Large Language Models (LLMs) is challenging, as hybrid training with long-context and short-context data often leads to workload imbalances. Existing works mainly use data packing to alleviate this issue, but fail to consider imbalanced attention computation and wasted communication overhead. This paper proposes Hierarchical Balance Packing (HBP), which designs a novel batch-construction method and training recipe to address those inefficiencies. In particular, the HBP constructs multi-level data packing groups, each optimized with a distinct packing length. It assigns training samples to their optimal groups and configures each group with the most effective settings, including sequential parallelism degree and gradient checkpointing configuration. To effectively utilize multi-level groups of data, we design a dynamic training pipeline specifically tailored to HBP, including curriculum learning, adaptive sequential parallelism, and stable loss. Our extensive experiments demonstrate that our method significantly reduces training time over multiple datasets and open-source models while maintaining strong performance. For the largest DeepSeek-V2 (236B) MoE model, our method speeds up the training by 2.4$\times$ with competitive performance. Codes will be released at https://github.com/ModelTC/HBP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。