无需检查点即可恢复大模型训练，降低通信开销。
All is Not Lost: LLM Recovery without Checkpoints
- 用邻近层加权平均替代故障层，避免额外存储和计算。
- 低中频故障下，训练时间比冗余计算快12%。
- 适合分布式训练场景，尤其对资源受限的团队友好。
在去中心化节点或本地实例上训练大语言模型可降低训练成本并促进模型民主化。但节点临时失效或调度策略导致部分模型（某些层）丢失是主要挑战。传统方法依赖检查点或冗余计算，即使无故障也带来显著通信与计算开销，且难以扩展至大规模模型。本文提出CheckFree，通过加权平均最近邻阶段来替代故障阶段，无需额外计算或存储。由于平均机制限制，仅适用于中间阶段故障。进一步提出CheckFree+，结合乱序流水线执行，可容忍首尾阶段崩溃：通过邻近阶段模拟其行为，实现恢复。对于嵌入与解码层，复制邻近阶段内容，仅需少量存储开销。我们在124M至1.5B参数的LLaMa模型上评估，低至中等故障率（5%-10%）下，CheckFree与CheckFree+在收敛耗时上均优于检查点与冗余计算，最高提升达12%。代码已开源：https://github.com/gensyn-ai/CheckFree。
原文摘要 · Abstract (English)
Training LLMs on decentralized nodes or on-spot instances, lowers the training cost and enables model democratization. The inevitable challenge here is the transient churns of nodes due to failures and the operator's scheduling policies, leading to losing parts of the model (some layers). The conventional approaches to recover from failures is to either use checkpointing, where periodically a copy of the entire model is sent to an additional storage, or redundant computation. These approaches yield significant communication and/or computation overhead even in non-failure cases and scale poorly in settings with large models. In this paper we propose CheckFree, an efficient recovery method where a failing stage is substituted by weighted averaging of the closest neighboring stages. In contrast to the state of the art, CheckFree requires no additional computation or storage. However, because of the nature of averaging neighbouring stages, it can only recover failures of intermediate stages. We further extend our method to CheckFree+ with out-of-order pipeline execution to tolerate crashes of the first and last stages. Thanks to out-of-order pipelining, behaviour of the first and last stages are mimicked by their neighboring ones, which allows CheckFree+ to recover them by copying the neighboring stages. To recover the (de-)embedding layers, CheckFree+ copies those layers in the neighboring stages, which requires relatively small storage overhead. We extensively evaluate our method on LLaMa models of model sizes from 124M to 1.5B with varying failure frequencies. In the case of low and medium failure rates (5-10%), CheckFree and CheckFree+ outperform both checkpointing and redundant computation in terms of convergence wall-clock time, achieving up to 12% improvement over redundant computation. Both of our proposals can be ran via our code available at: https://github.com/gensyn-ai/CheckFree
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。