arXiv:2509.26246cs.AI2025-09

通过细粒度分片打包,解决大模型训练中的长度不均问题。

SlimPack: Fine-Grained Asymmetric Packing for Balanced and Efficient Variable-Length LLM Training

  • 将样本拆分为细粒度片段,缓解内存与通信瓶颈。
  • 采用非对称分区策略,实现前向与反向计算的负载均衡。
  • 实测训练吞吐提升2.8倍,兼顾效率与资源利用率。

大型语言模型(LLMs)的高效分布式训练严重受制于上下文长度的极端差异。这种数据异质性在传统打包策略和前后向计算成本不对称的放大下,导致严重的负载不均衡和硬件利用率低下。现有方案虽试图缓解此问题,但常以牺牲内存或通信效率为代价。为此,我们提出SlimPack,通过将样本分解为细粒度切片,从根本上重构数据打包与调度。该切片级分解立即缓解了关键的内存与通信瓶颈,将大而波动的工作负载转化为一系列小而可控的单元。这一灵活性被用于核心创新——非对称分区,其组装出针对前向与反向计算不同需求优化的均衡调度单元。由两阶段求解器与高保真模拟器协同控制,SlimPack全面解决了所有并行维度的不平衡问题。大量实验表明,相比基线,SlimPack实现了高达2.8倍的训练吞吐量提升,打破了传统权衡,同时实现优异的平衡性与高资源效率。

原文摘要 · Abstract (English)

The efficient distributed training of Large Language Models (LLMs) is severely hampered by the extreme variance in context lengths. This data heterogeneity, amplified by conventional packing strategies and asymmetric forward-backward costs, leads to critical inefficiencies such as cascading workload imbalances and severe hardware underutilization. Existing solutions attempt to mitigate these challenges, but often at the expense of memory or communication efficiency. To address these challenges, we introduce SlimPack, a framework that fundamentally rethinks data packing and scheduling by decomposing samples into fine-grained slices. This slice-level decomposition immediately mitigates critical memory and communication bottlenecks by transforming large, volatile workloads into a stream of smaller, manageable units. This flexibility is then harnessed for our core innovation, Asymmetric Partitioning, which assembles balanced scheduling units uniquely optimized for the different demands of the forward and backward passes. Orchestrated by a two-phase solver and a high-fidelity simulator, SlimPack holistically resolves imbalances across all parallel dimensions. Extensive experiments demonstrate that SlimPack achieves up to a $2.8\times$ training throughput improvement over baselines, breaking the conventional trade-off by delivering both superior balance and high resource efficiency.

大模型训练负载均衡高效推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。