用实测数据揭示学术界预训练模型的可行路径与成本权衡
$100K or 100 Days: Trade-offs when Pre-Training with Academic Resources
- 构建基准测试工具，量化不同学术级显卡的训练耗时
- 在4块GPU上18天完成Pythia-1B模型复现，仅需原方案1/3的计算量
- 为资源有限的研究者提供可落地的训练策略和成本参考
预训练通常需要大量算力，而学术研究者普遍资源匮乏，因此常被认为无法进行模型预训练。本文通过调查学术研究者的可用算力，并实测其在实际硬件上的训练时间，提出一个用于衡量特定显卡上预训练耗时的基准。我们对多种模型与学术级GPU进行了实验，共消耗2000 GPU小时。结果显示：例如原版Pythia-1B在64块GPU上训练3天，但可在4块GPU上以相同超参数运行18天完成复现，仅需原方案1/3的GPU天数。最后我们进行成本效益分析，明确价格与训练时间之间的权衡关系。我们认为该基准将帮助学术研究者更有效地在更大规模数据上训练大模型。代码已完全开源：https://github.com/apoorvkh/academic-pretraining。
原文摘要 · Abstract (English)
Pre-training is notoriously compute-intensive and academic researchers are notoriously under-resourced. It is, therefore, commonly assumed that academics can't pre-train models. In this paper, we seek to clarify this assumption. We first survey academic researchers to learn about their available compute and then empirically measure the time to replicate models on such resources. We introduce a benchmark to measure the time to pre-train models on given GPUs and also identify ideal settings for maximizing training speed. We run our benchmark on a range of models and academic GPUs, spending 2,000 GPU-hours on our experiments. Our results reveal a brighter picture for academic pre-training: for example, although Pythia-1B was originally trained on 64 GPUs for 3 days, we find it is also possible to replicate this model (with the same hyper-parameters) in 3x fewer GPU-days: i.e. on 4 GPUs in 18 days. We conclude with a cost-benefit analysis to help clarify the trade-offs between price and pre-training time. We believe our benchmark will help academic researchers conduct experiments that require training larger models on more data. We fully release our codebase at: https://github.com/apoorvkh/academic-pretraining.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。