提出SMET方法，让大模型稀疏训练更稳定高效。
Memory-Efficient LLM Training with Dynamic Sparsity: From Stability to Practical Scaling
- 用优化器预热和密度自适应学习率提升稀疏训练稳定性
- 在不牺牲性能前提下，内存消耗降低至密集训练的30%以下
- 适合资源有限但需训练大规模语言模型的研究者
动态稀疏训练（DST）虽能提升深度神经网络的训练与推理效率，但在大语言模型训练中易出现优化不稳定问题，表现为拓扑更新后损失突增。本文发现，使用标准Adam优化器会引发新恢复参数的冷启动问题，导致更新幅度过大、训练过程失稳。为此提出稀疏内存高效训练（SMET），通过优化器预热和密度感知学习率缩放实现稳定训练，并仅存储活跃参数的梯度与优化器状态，显著降低内存占用。理论分析表明，SMET改善了参数更新行为，提升了优化稳定性。大量实验验证，SMET可实现稳定、可扩展且内存高效的稀疏预训练，为稀疏训练成为密集训练的实际替代方案铺平道路。代码已开源：https://github.com/QiaoXiao7282/SMET。
原文摘要 · Abstract (English)
Dynamic Sparse Training (DST) offers a promising paradigm for improving the training and inference efficiency of deep neural networks; however, we find that in large language model training, DST can suffer from optimization instability, manifested as loss spikes after topology updates. In this work, we show that the naive use of standard Adam-based optimizers leads to a cold-start issue for newly regrown parameters, resulting in excessively large updates and disrupted training dynamics. To address this issue, we propose Sparse Memory-Efficient Training (SMET), which stabilizes DST with optimizer warm-up and improves training progress through density-aware learning-rate scaling. SMET further reduces memory consumption by storing gradients and optimizer states only for active parameters. We provide a theoretical analysis of the update behaviors under SMET, showing improved optimization stability. Extensive experiments demonstrate that SMET enables stable, scalable, and memory-efficient sparse pre-training of LLMs, paving the way for sparse training as a practical alternative to dense training. Our code is publicly available at: https://github.com/QiaoXiao7282/SMET.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。