自动生成复杂计算机任务，低成本构建高质量训练数据
AgentSynth: Scalable Task Generation for Generalist Computer-Use Agents
- 利用信息不对称设计分步任务，组合后形成高难度长时序任务
- 生成超6000个真实任务，6级难度下模型成功率仅4%
- 每条轨迹成本仅0.6美元，远低于人工标注
我们提出AgentSynth，一种可扩展且成本低廉的自动化流水线，用于为通用计算机使用智能体生成高质量任务与轨迹数据集。通过利用信息不对称，AgentSynth构造出生成时简单但组合后极具挑战性的子任务，从而创建了超过6000个多样化且真实的任务。其核心优势在于可通过调整子任务数量精确调控任务复杂度。实验表明，最先进大模型智能体在难度等级1时成功率18%，而到等级6时骤降至4%，凸显该基准的难度与区分能力。此外，本方法平均每个轨迹成本仅0.60美元，相比人工标注低几个数量级。代码与数据已开源：https://github.com/sunblaze-ucb/AgentSynth
原文摘要 · Abstract (English)
We introduce AgentSynth, a scalable and cost-efficient pipeline for automatically synthesizing high-quality tasks and trajectory datasets for generalist computer-use agents. Leveraging information asymmetry, AgentSynth constructs subtasks that are simple during generation but significantly more challenging when composed into long-horizon tasks, enabling the creation of over 6,000 diverse and realistic tasks. A key strength of AgentSynth is its ability to precisely modulate task complexity by varying the number of subtasks. Empirical evaluations show that state-of-the-art LLM agents suffer a steep performance drop, from 18% success at difficulty level 1 to just 4% at level 6, highlighting the benchmark's difficulty and discriminative power. Moreover, our pipeline achieves a low average cost of $0.60 per trajectory, orders of magnitude cheaper than human annotations. Our code and data are available at https://github.com/sunblaze-ucb/AgentSynth
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。