arXiv:2606.22883cs.AI2026-06被引 3

构建可验证的终端任务生成引擎,提升训练数据质量与模型效率。

CLI-Universe: Towards Verifiable Task Synthesis Engine for Terminal Agents

论文配图:CLI-Universe: Towards Verifiable Task Synthesis Engine for Terminal Agents
图 1 · 摘自论文原文
  • 按能力维度组合生成候选任务,再通过真实技术资料验证可行性。
  • 仅保留1/3通过多阶段可执行验证的任务,确保任务真实且有挑战性。
  • 用6000条高质量轨迹微调模型,性能超越更大规模模型。

尽管基于大模型的终端代理已展现出潜力,但高质量可执行训练数据的匮乏仍是关键瓶颈。现有合成流程通常仅对表面内容进行改造,导致指令模糊、执行路径浅显、测试脆弱,学习信号弱。为此,我们提出CLI-Universe,一个基于原则的任务合成引擎,用于构建终端代理任务。该引擎通过在多维能力分类体系(领域、技能类型、能力、工程支柱)中采样组合生成候选任务,并基于真实技术材料进行证据引导的深度研究以实现任务锚定。为确保严格监督,经验证的蓝图被实例化为Docker环境,并经过多阶段可执行验证流程:基于评分标准的测试构建、提示条件过滤和严格失败转成功检测。整个流程中约三分之二的候选任务被剔除,仅保留真正可验证且非平凡挑战性的任务。为验证框架有效性,我们构建了一个精炼数据集CLI-Universe-6K,包含6,000条轨迹。将Qwen3-32B在该数据集上微调后,在Terminal-Bench 2.0上达到33.4%准确率,创下开源数据下32B参数以下模型的新纪录,优于多个规模大一个数量级的模型,证明了结构化高保真合成带来的显著数据效率优势。

原文摘要 · Abstract (English)

While recent LLM-based terminal agents have demonstrated promising capabilities, the scarcity of high-quality, executable training data remains a critical bottleneck. Existing synthesis pipelines typically scale by retrofitting surface-level artifacts into tasks, frequently yielding ambiguous instructions, shallow execution paths, and brittle tests that provide weak learning signals. To overcome this, we introduce CLI-Universe, a principled synthesis engine that constructs terminal-agent tasks. CLI-Universe generates candidate tasks by sampling combinations across a multi-dimensional capability taxonomy (domain, skill type, capability, and engineering pillar), then grounds each candidate through evidence-guided deep research over real-world technical materials. To ensure rigorous supervision, validated blueprints are instantiated into Dockerized environments and subjected to a multi-stage executable verification pipeline featuring rubric-gated test construction, hint-conditional filtering, and strict fail-to-pass checking. Across the full pipeline, from candidate generation to verification, approximately two-thirds of candidates are discarded, retaining only those that are genuine, verifiable, and non-trivially challenging. To validate our framework, we instantiate a highly distilled dataset of 6,000 trajectories called CLI-Universe-6K. Remarkably, fine-tuning Qwen3-32B on CLI-Universe-6K achieves 33.4% on Terminal-Bench 2.0. This sets a new state-of-the-art for models trained on open-source data at or below 32B parameters, and outperforms several models an order of magnitude larger, demonstrating the profound data efficiency of structured, high-fidelity synthesis.

任务合成终端代理数据质量可验证性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。