针对大模型训练故障，提出分层存档系统实现快速恢复。
TierCheck: Tiered Checkpointing for Fault Tolerance in Large Language Model Training
- 按故障类型分层存储：本地、对端内存存轻量差分快照，远程存重载基线
- 400亿参数模型下，存档时间低于10秒，支持高频存档
- 自动适配集群故障特征，适合大规模分布式训练场景
大语言模型训练常因从单卡崩溃到全集群中断等异构故障而中断。现有存档系统依赖单一存储层级，导致状态保存开销与恢复速度难以兼顾。我们提出 TierCheck，一种感知集群的分层存档系统，将存储位置与故障异构性对齐。该系统采用三层设计：在本地和对端内存中维护轻量级差分快照以实现快速局部恢复，同时异步将重型基线快照迁移至远程持久化存储。系统还确保跨层级严格全局一致性，且不阻塞训练过程，支持故障时的快速集群感知恢复。在最大达400亿参数的模型上评估显示，TierCheck 实现低训练开销，端到端存档时间低于10秒，支持高频存档，最终在低开销持久化与快速恢复间取得最优平衡。
原文摘要 · Abstract (English)
Large Language Model (LLM) training is frequently interrupted by a heterogeneous spectrum of failures, from common GPU crashes to catastrophic cluster-wide outages. Existing checkpointing systems rely on monolithic, single-tier storage backend, forcing a trade-off between state-saving overhead and recovery speed. We propose TierCheck, a cluster-aware tiered checkpointing system that aligns storage placement with failure heterogeneity. TierCheck adopts a three-tier design that maintains lightweight differential checkpoints in local and peer memory for fast localized recovery, while asynchronously migrating heavyweight base checkpoints to remote persistent storage. It also ensures strict global consistency across tiers without stalling training, and achieves fast cluster-aware checkpoint restoration during recovery. Evaluations on models up to 40 billion parameters show that TierCheck achieves low training overhead, reduces end-to-end checkpointing time to under 10s, and supports high-frequency checkpointing, ultimately striking an optimal balance between low-overhead persistence and fast recovery.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。