解决长文本训练中计算负载不均问题，提升效率与模型精度。
SparseBalance: Load-Balanced Long Context Training with Dynamic Sparse Attention
- 动态调整注意力稀疏度，缓解计算延迟
- 通过分组策略实现粗粒度负载均衡，提升训练速度1.33倍
- 适合大规模长文本模型训练场景，兼顾性能与效率
稀疏注意力虽缓解了长上下文大模型训练的计算瓶颈，但其分布式训练过程在序列长度和稀疏敏感性上存在严重异质性，导致负载失衡与模型精度下降。现有方法多聚焦单一问题，未能系统协同优化。为此，我们提出SparseBalance——一种算法与系统协同设计框架，利用稀疏性和序列异质性，联合优化模型精度与系统效率。首先，提出基于工作量感知的动态稀疏度调节，通过双向调整消除慢节点，利用空闲资源提升精度；其次，提出稀疏度感知的批处理策略，实现粗粒度负载平衡，与动态调节互补。实验表明，SparseBalance在LongBench基准上实现最高1.33倍端到端加速，同时长文本能力提升0.46%。
原文摘要 · Abstract (English)
While sparse attention mitigates the computational bottleneck of long-context LLM training, its distributed training process exhibits extreme heterogeneity in both \textit{1)} sequence length and \textit{2)} sparsity sensitivity, leading to a severe imbalance problem and sub-optimal model accuracy. Existing algorithms and training frameworks typically focus on single issue, failing to systematically co-optimize these two problems. Therefore, we propose SparseBalance, a novel algorithm-system co-design framework, which exploits the sparsity and sequence heterogeneity to optimize model accuracy and system efficiency jointly. First, we propose workload-aware dynamic sparsity tuning, which employs a bidirectional sparsity adjustment to eliminate stragglers and exploit inherent bubbles for free accuracy. Second, we propose a sparsity-aware batching strategy to achieve coarse-grained balance, which complements dynamic sparsity tuning. Experimental results demonstrate that SparseBalance achieves up to a 1.33$\times$ end-to-end speedup while still improving the long-context capability by 0.46\% on the LongBench benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。