Litespark提升大模型训练速度与能效，30B模型提速6倍、省电超80%。
Litespark Technical Report: High-Throughput, Energy-Efficient LLM Training Framework
- 优化Transformer的注意力与MLP层，提升计算效率。
- 30B模型在多节点H200上训练速度提升6倍，能耗降低83%。
- 兼容标准Transformer，适用于预训练与微调等阶段。
大语言模型（LLM）训练面临训练时间长、能耗高的问题，现代模型需数月计算和数百兆瓦时电力。为此，我们提出Litespark，一种新型预训练框架，通过针对性优化Transformer注意力与MLP层，结合架构改进与算法优化，在保持与标准Transformer兼容的前提下，最大化模型浮点运算利用率（MFU）。在3B和30B参数的Llama模型上，使用SlimPajama-627B数据集进行综合基准测试显示：多节点H200 GPU集群中，训练吞吐量提升2至6倍，能耗降低55%至83%。该优化方法对模型和硬件均无依赖，可广泛应用于各类Transformer架构，并延伸至后训练阶段，如监督微调与直接偏好优化。
原文摘要 · Abstract (English)
Training Large Language Models (LLMs) is plagued by long training times and massive energy consumption, with modern models requiring months of computation and gigawatt-hours of electricity. In light of these challenges,we introduce Litespark, a novel pre-training framework that addresses these inefficiencies through targeted optimizations to transformer attention and MLP layers. Our approach combines architectural improvements with algorithmic enhancements to maximize Model FLOPs Utilization (MFU) while maintaining compatibility with standard transformer implementations. Comprehensive benchmarking on 3B and 30B parameter Llama models using the SlimPajama-627B dataset demonstrates substantial performance gains: 2x-6x training throughput improvement and $55\%-83$% energy consumption reduction across multi-node H200 GPU clusters. These optimizations are model- and hardware-agnostic, enabling broad applicability across transformer architectures and extending to post-training phases including supervised fine-tuning and direct preference optimization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。