Tmax用简单方法让小模型在终端任务上超越大模型
Tmax: A simple recipe for terminal agents
- 用新分类法生成多样化终端环境数据，控制难度并提升多样性
- 仅90亿参数就达27%准确率，超过更大规模旧模型
- 开源数据、模型和代码，为学术研究提供强力基准
终端使用型智能体已成为语言模型最主流的下游应用。尽管广泛应用，但基于强化学习的训练研究仍较少，原因包括评测困难、数据匮乏及缺乏简易基线方案。本文提出Tmax，当前最强的开源强化学习训练方案，使开放数据方案更接近前沿水平。尽管方法简单，其仅用90亿参数即在Terminal-Bench 2.0上达到27%准确率，优于此前更大模型的表现。我们通过结合难度控制、角色设定与验证器多样性构建新型数据生成分类法，可低成本生成大量用于强化学习与监督微调的终端环境数据。所发布的数据集规模超过此前公开数据集的2.5倍。我们使用该数据训练开放权重模型，采用仅基于结果反馈的简单强化学习策略，并将数据、模型与代码全部开源，为未来学术研究提供强大基准，详见https://github.com/hamishivi/tmax。
原文摘要 · Abstract (English)
Terminal-using agents have quickly become the most popular downstream application of language models (LMs). Despite their prevalence, relatively little academic work has examined RL-based training of these models, likely due to difficult benchmarks, a lack of data, and a lack of simple baseline recipes. We present Tmax, the strongest open RL recipe for terminal agents to date, bringing open data recipes closer to the frontier. While simple, our recipe achieves 27\% on Terminal-Bench 2.0 with only 9B parameters, outperforming much larger models from prior work. Concretely, we generate data using a novel taxonomy, combining difficulty control, personas, and verifier diversification, which allows us to cheaply generate large amounts of terminal environments for RL and SFT training. We open-source our terminal dataset, which is over 2.5x larger than previously released terminal-agent datasets. We then train open-weight models using RL with our data, using a simple, outcome-only recipe. We release our data, models, and code as a strong baseline for future open academic work on terminal agents at https://github.com/hamishivi/tmax.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。