arXiv:2608.14635cs.DCcs.LG2026-08

提升大模型智能体强化学习训练的容错效率,避免失败导致工作丢失。

Belayer: Efficient Fault Tolerance for LLM Agentic RL Training

论文配图:Belayer: Efficient Fault Tolerance for LLM Agentic RL Training
图 1 · 摘自论文原文
  • 为推理引擎和环境分别设计轻量级故障恢复机制。
  • 单次任务恢复速度比冷启动快42倍,环境故障恢复提速1.5至3.5倍。
  • 适用于长周期、高复杂度的大模型智能体训练场景。

大语言模型(LLM)智能体在长周期、沙盒环境中通过强化学习进行训练正日益普遍。与传统强化学习不同,智能体强化学习将高耗显存的生成引擎与有状态的环境容器结合,其动作可能产生可见副作用,如文件修改、命令执行和依赖安装。单个轨迹可跨越多轮生成与环境交互,任一组件故障都可能导致已完成工作丢失或模型陷入与上下文不一致的环境状态。现有系统缺乏高效且正确的分布式执行容错机制。本文提出Belayer,一种高效的LLM智能体强化学习容错系统。针对局部工作节点故障,Belayer为每个预初始化的影子工作节点配备选择性显存复用协议,保留独立拥有的权重与原始KV缓存分配,在健康检查后重置本地状态,并从日志中重建请求特定的KV内容。针对环境故障,引入全状态检查点与恢复机制,联合捕获并还原容器文件系统与运行时状态,并协调恢复后的环境与LLM上下文以保持前缀一致性。采用自适应策略,在预测间隔足够长时,将全状态检查点操作与自然的LLM推理空档重叠。实测表明,无故障训练下开销极低,工作节点恢复时间相比全引擎冷启动最高降低42倍,环境故障恢复速度提升1.5至3.5倍。

原文摘要 · Abstract (English)

Large language model (LLM) agents are increasingly trained with reinforcement learning in long-horizon, sandboxed environments. Unlike conventional RL, agentic RL couples GPU-intensive rollout engines with stateful environment containers whose actions may produce visible side effects, such as file edits, command execution, and dependency installation. A single trajectory can span many rounds of gen- eration and environment interaction, so a component failure can discard completed work or expose the model to an environment state that is inconsistent with its context. However, existing systems lack efficient and correct recovery mechanisms for this distributed execution model. This paper presents Belayer, an efficient fault-tolerant system for LLM agentic RL training. Belayer handles failures in both rollout engines and environment execution while targeting low failure-free overhead. For scoped worker-local rollout failures, Belayer equips each pre-initialized shadow worker with a selective GPU-state reuse protocol that retains independently owned weights and raw KV-arena allocations after owner and GPU health checks, reinitializes worker-local state, and rebuilds request-specific KV contents from logged token prefixes. For environment failures, Belayer introduces full checkpoint and full restore to jointly capture and restore container file-system and runtime state, and coordinates the recovered environment with the LLM context to preserve prefix consistency. An adaptive policy opportunistically overlaps full-state checkpointing with natural LLM inference bubbles when the predicted interval is long enough. Empirical results show low measured overhead during failure-free training, a worker-recovery-time reduction of up to 42 times faster compared with a full engine cold start, and 1.5 to 3.5 times faster recovery from environment failures.

大模型强化学习容错智能体

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。