用递归验证法低成本生成复杂长时序任务,提升智能体训练效果。
Recursive Synthesis for Long-Horizon Terminal Tasks

- 从可信种子任务出发,递归扩展并验证新任务流程。
- 每任务成本仅0.05美元,15轮生成3.7万条任务,难度持续上升。
- 适用于需要长序列决策能力的智能体训练与评估场景。
高质量长时序终端智能体训练数据成本高昂,通常每项任务需数百至数千美元,因指令、环境、参考解与验证器必须保持一致。人工编写难扩展,直接用大语言模型生成常破坏依赖关系。本文提出递归验证合成框架RST,可规模化生成长时序终端任务。从已验证种子任务出发,RST扩展参考解,重对齐验证器与指令,于新沙盒中验证结果,并将通过的任务作为下一轮种子。经十五轮递归,共生成37,484条合成任务,平均成本约0.05美元/任务。任务难度显著提升:参考解中位长度由67行增至374行,执行命令中位数由40增至244,DeepSeek-V4-Pro pass@4从90%降至2.5%。为验证训练价值,我们采集拒绝采样后的Qwen3.5轨迹用于监督微调。微调后,Qwen3.5-27B与Qwen3.5-122B-A10B在Terminal-Bench 2、Terminal-Bench Hard和Long-Horizon Terminal Bench上性能最高提升10分;使用代理PPO进一步将Qwen3.5-27B提升至49.44%、32.00%、22.07%,相较基线分别提升20.0%、41.2%、21.9%。且15轮后合成效率与验证率仍稳定,显示该过程可继续扩展。
原文摘要 · Abstract (English)
High-quality long-horizon training data for terminal agents is expensive to produce, often costing hundreds to thousands of dollars per task, because each task must keep the instruction, environment, reference solution, and verifier mutually consistent. Human authoring does not scale, and direct generation with large language models (LLMs) often breaks these dependencies. We present Recursive Synthetic Terminal Tasks (RST), a recursive verified synthesis framework for constructing long-horizon terminal-agent tasks at scale. Starting from verified seed tasks, RST extends the reference solution, realigns the verifier and instruction to the new workflow, validates the result in a fresh sandbox, and reuses accepted tasks as seeds for subsequent rounds. Across fifteen recursive rounds, RST produces 37,484 synthesized terminal-agent tasks at roughly $0.05 per task. Task difficulty increases substantially over rounds: the median reference solution grows from 67 to 374 lines, the median number of executed commands grows from 40 to 244, and DeepSeek-V4-Pro pass@4 drops from 90% at R1 to 2.5% at R15. To demonstrate training utility, we collect rejection-sampled Qwen3.5 trajectories on the synthesized tasks and use them for supervised fine-tuning. Fine-tuning on these trajectories improves Qwen3.5-27B and Qwen3.5-122B-A10B by up to 10 points on Terminal-Bench 2, Terminal-Bench Hard, and Long-Horizon Terminal Bench, while agentic PPO lifts Qwen3.5-27B to 49.44%, 32.00%, and 22.07% on the three benchmarks, corresponding to relative gains of 20.0%, 41.2%, and 21.9% over the base model. Moreover, after 15 rounds, the recursion shows no ceiling: synthesis yield and validation rates remain stable as difficulty keeps climbing, indicating that the process can continue well beyond the scale reported here.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。