STEP通过分阶段高效微调，让大模型预训练内存减少超一半。
STEP: Staged Parameter-Efficient Pre-training for Large Language Models
- 分阶段引入参数高效技术，逐步扩展模型规模
- 最大内存需求降低53.9%，性能与原方法相当
- 适合资源有限但需训练大模型的研究者
大语言模型（LLM）预训练面临参数量巨大带来的内存挑战。我们提出分阶段参数高效预训练（STEP），将参数高效微调技术与模型渐进式增长相结合。在多种规模的LLM上进行实验，结果表明，与传统预训练相比，STEP可将最大内存需求降低高达53.9%，同时保持相当的性能表现。此外，经过指令微调后，STEP训练的模型在下游任务上的表现与标准预训练模型相当。
原文摘要 · Abstract (English)
Pre-training large language models (LLMs) faces significant memory challenges due to the large size of model parameters. We introduce STaged parameter-Efficient Pre-training (STEP), which integrates parameter-efficient tuning techniques with model growth. We conduct experiments on pre-training LLMs of various sizes and demonstrate that STEP achieves up to a 53.9% reduction in maximum memory requirements compared to vanilla pre-training while maintaining equivalent performance. Furthermore, we show that the model by STEP performs comparably to vanilla pre-trained models on downstream tasks after instruction tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。