用失败案例生成训练任务，让大模型学会更可靠地使用工具。
SENTINEL: Failure-Driven Reinforcement Learning for Training Tool-Using Language Model Agents
- 通过分析失败轨迹，自动生成针对性训练任务
- 在零售任务上成功率从66.4%提升至74.9%
- 适合想提升工具使用能力的AI Agent开发者
语言模型代理通过多轮工具使用解决现实任务的能力日益增强。然而，训练可靠的工具使用代理仍具挑战性。尽管强化学习能基于自身环境交互改进代理，但其效果高度依赖训练任务分布。若任务在训练前固定，任务分布会随策略能力演变而逐渐失配，导致大量采样浪费在无信息的任务上。我们提出SENTINEL，一种以失败驱动的强化学习框架，将求解器的回溯失败转化为目标训练任务。SENTINEL采用控制器-提议者-求解器循环：控制器分析失败轨迹并总结重复错误模式，提议者生成针对这些弱点的可执行任务，求解器在目标任务上进行训练。在Tau2-Bench Retail数据集上，使用Qwen3-4B-Thinking-2507模型，SENTINEL将Pass^1从66.4%提升至74.9%，并在通用合成任务的Pass^k指标上优于传统RL方法。结果表明，模型失败是提升工具使用语言模型代理的有效且可扩展的训练信号来源。
原文摘要 · Abstract (English)
Language model agents are increasingly effective in solving realistic tasks through multi-turn tool use. However, training reliable tool-using agents remains challenging in practice. While reinforcement learning provides an on-policy paradigm for improving agents from their own environment interactions, its effectiveness depends heavily on the training task distribution. When tasks are fixed before training, the task distribution can become increasingly mismatched with the policy's evolving capabilities, causing many rollouts to be spent on uninformative tasks. We propose SENTINEL, a failure-driven reinforcement learning framework that turns the Solver's rollout failures into targeted training tasks. SENTINEL follows a Controller--Proposer--Solver loop: the Controller analyzes failed trajectories and summarizes recurring error patterns, the Proposer generates executable tasks that stress these weaknesses, and the Solver is trained on the targeted tasks. On Tau2-Bench Retail with Qwen3-4B-Thinking-2507, SENTINEL improves Pass\^{}1 from 66.4 to 74.9 and outperforms RL on general synthetic tasks across Pass\^{}k metrics. These results demonstrate that model failures provide an effective and scalable source of targeted training signal for improving tool-using language model agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。