arXiv:2601.16443cs.LGcs.CL2026-01被引 27

用自动生成任务的流水线训练终端智能体,效果远超传统方法。

Endless Terminals: Scaling RL Environments for Terminal Agents

  • 自动生成3255个终端任务,覆盖文件、日志、数据等操作
  • 无复杂工具下,模型准确率提升至59.0%,超越人类标注基准
  • 适合研究强化学习环境扩展与简单智能体训练的学者

环境是自进化智能体的瓶颈。当前终端评测集仅用于评估,而非训练;强化学习需要可扩展的训练流水线,而不仅是数据集。我们提出Endless Terminals,一个完全自动化的流水线,无需人工标注即可程序化生成终端使用任务。该流水线包含四个阶段:生成多样任务描述、构建并验证容器化环境、生成完成测试、筛选可解性任务。由此获得3255个任务,涵盖文件操作、日志管理、数据处理、脚本编写和数据库操作。使用基础PPO算法,以二值化回合奖励和极简交互循环训练智能体:无检索、无多智能体协作、无专用工具。尽管结构简单,训练后的模型表现显著提升:在保留开发集上,Llama-3.2-3B从4.0%提升至18.2%,Qwen2.5-7B从10.7%升至53.3%,Qwen3-8B-openthinker-sft从42.6%增至59.0%。这些改进可迁移至人工标注基准:在TerminalBench 2.0上,Llama-3.2-3B从0.0%提至2.2%,Qwen2.5-7B从2.2%至3.4%,Qwen3-8B-openthinker-sft从1.1%至6.7%,均优于采用更复杂代理架构的模型。结果表明,当环境可扩展时,简单强化学习亦能成功。

原文摘要 · Abstract (English)

Environments are the bottleneck for self-improving agents. Current terminal benchmarks were built for evaluation, not training; reinforcement learning requires a scalable pipeline, not just a dataset. We introduce Endless Terminals, a fully autonomous pipeline that procedurally generates terminal-use tasks without human annotation. The pipeline has four stages: generating diverse task descriptions, building and validating containerized environments, producing completion tests, and filtering for solvability. From this pipeline we obtain 3255 tasks spanning file operations, log management, data processing, scripting, and database operations. We train agents using vanilla PPO with binary episode level rewards and a minimal interaction loop: no retrieval, multi-agent coordination, or specialized tools. Despite this simplicity, models trained on Endless Terminals show substantial gains: on our held-out dev set, Llama-3.2-3B improves from 4.0% to 18.2%, Qwen2.5-7B from 10.7% to 53.3%, and Qwen3-8B-openthinker-sft from 42.6% to 59.0%. These improvements transfer to human-curated benchmarks: models trained on Endless Terminals show substantial gains on held out human curated benchmarks: on TerminalBench 2.0, Llama-3.2-3B improves from 0.0% to 2.2%, Qwen2.5-7B from 2.2% to 3.4%, and Qwen3-8B-openthinker-sft from 1.1% to 6.7%, in each case outperforming alternative approaches including models with more complex agentic scaffolds. These results demonstrate that simple RL succeeds when environments scale.

强化学习智能体训练自动生成终端任务

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。