针对大模型强化学习训练中的故障,实现按角色隔离恢复,大幅减少重启开销。
Role-Based Fault Tolerance System for LLM RL Post-Training
- 将训练、推理等角色分离,故障仅影响对应角色。
- 在256卡集群上故障率10%时,有效训练时间比率达80%以上。
- 支持快速恢复与动态通信,适合大规模强化学习训练场景。
大语言模型的强化学习后训练已广泛用于提升推理与工具使用能力。然而,该过程同时包含训练与推理负载,易受双方故障影响。现有容错框架仅覆盖训练或推理,未充分利用异步执行的优化潜力。本文提出基于角色的故障隔离机制,将训练器、回滚和管理等角色视为独立分布式任务。在RobustRL系统中,故障发生时仅恢复失败角色并重新连接至存活节点,避免全任务重启带来的回滚重放与初始化延迟。系统具备三项核心能力:(1)角色感知监控,区分真实故障与正常行为,减少误报与延迟;(2)非中断式重启:训练器通过热备状态延续轨迹生成,回滚节点可独立替换而不中断任务;(3)动态通信重构:采用基于UCX的点对点通信,实现恢复角色的即时权重同步。在256卡集群上运行Qwen3-8B-Math任务,故障注入频率为10%时,RobustRL的有效训练时间比率达80%以上,较ByteRobust的60%显著提升,端到端训练速度加快8.4%–17.4%。
原文摘要 · Abstract (English)
RL post-training for LLMs has been widely scaled to enhance reasoning and tool-using capabilities. However, RL post-training interleaves training and inference workloads, exposing the system to faults from both sides. Existing fault tolerance frameworks for LLMs target either training or inference, leaving the optimization potential in the asynchronous execution unexplored for RL. Our key insight is role-based fault isolation so the failure in one machine does not affect the others. We treat trainer, rollout, and other management roles in RL training as distinct distributed sub-tasks. Instead of restarting the entire RL task in ByteRobust, we recover only the failed role and reconnect it to living ones, thereby eliminating the full-restart overhead including rollout replay and initialization delay. We present RobustRL, the first comprehensive robust system to handle GPU machine errors for RL post-training Effective Training Time Ratio improvement. (1) \textit{Detect}. We implement role-aware monitoring to distinguish actual failures from role-specific behaviors to avoid the false positive and delayed detection. (2) \textit{Restart}. For trainers, we implement a non-disruptive recovery where rollouts persist state and continue trajectory generation, while the trainer is rapidly restored via rollout warm standbys. For rollout, we perform isolated machine replacement without interrupting the RL task. (3) \textit{Reconnect}. We replace static collective communication with dynamic, UCX-based (Unified Communication X) point-to-point communication, enabling immediate weight synchronization between recovered roles. In an RL training task on a 256-GPU cluster with Qwen3-8B-Math workload under 10\% failure injection frequency, RobustRL can achieve an ETTR of over 80\% compared with the 60\% in ByteRobust and achieves 8.4\%-17.4\% faster in end-to-end training time.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。