用程序化生成的夺旗任务训练和评估终端操作能力。
unix-ctf: Procedural Environments for Unix-Competence Reinforcement Learning
- 通过LLM辅助生成可复用的隐藏与查找脚本，构建真实可用的Unix技能训练环境。
- 微调后模型在15项技能测试中解题率从11.6%提升至43.6%，取证任务提升33个百分点。
- 适合研究终端智能体、系统安全或自动化运维的开发者与研究人员。
Unix能力是指将命令行与操作系统原语作为第一性工具使用，而非仅通过终端编写程序。现有终端基准常模糊这一区别：精通Python但弱于Unix的求解器仍能通过大量题目。本文提出unix-ctf，一个用于壳层智能体的程序化夺旗任务生成器。每个任务在新Linux容器中利用单一Unix特性隐藏一个短令牌（如flag(a3b1c9...)），智能体需恢复它。任务由LLM辅助合成流水线生成：候选藏匿方法经重写为参数化隐藏-查找脚本对，并通过双向合约过滤——隐藏脚本不得在磁盘留下明文痕迹，查找脚本必须在新目录中成功恢复。因LLM仅负责种植与恢复步骤（容器、布局、评分框架固定），该流程将750次尝试中的656次转化为可移植、可复用变体（87.5%）。相比之下，复现Endless Terminals全容器生成方法仅17.4%通过相同检验。这656个变体归约为155种独立技术。使用GRPO与LoRA对Qwen3-8B进行微调，在15项技能的多家族保留测试集（n=225）上解题率从11.6%升至43.6%，重新分配了模型对InterCode-CTF任务的解决能力，并在取证任务上取得+33百分点提升，最终达到InterCode-CTF 32/100的成绩。结果表明，Unix能力是可分离、可训练且应直接评估的，不应被混入编程式终端使用中。
原文摘要 · Abstract (English)
Unix competence is the ability to use shell and operating-system primitives as first-class tools, not merely to write programs through a terminal. Current terminal benchmarks tend to blur this distinction: a solver fluent in Python but weak in Unix can pass a substantial fraction of Terminal-Bench 2.0, while the reverse skill profile is rarely exercised. We make the distinction operational and build a training surface for the Unix component. unix-ctf is a procedural generator of capture-the-flag tasks for shell agents. Each task hides a short token (a flag of the form flag(a3b1c9...)) inside a fresh Linux container using a single Unix feature, and the agent must recover it. Tasks are produced by an LLM-assisted synthesis pipeline that generates candidate hiding techniques, rewrites them into parameterized hide-and-find script pairs, and filters them with a bidirectional contract: the hide script must leave no plaintext trace of the flag on disk, and the find script must recover the flag in a fresh directory. Because the LLM only writes the planting and recovery steps (the container, layout, and grading harness are fixed), the pipeline lands 656 of 750 raw attempts as portable, reusable variants (87.5\%). Our reproduction of Endless Terminals' full-container-generation approach lands only 17.4\% under the same checks. The 656 variants canonicalize to 155 distinct techniques. Fine-tuning Qwen3-8B with LoRA using GRPO on this surface lifts solve rate from 11.6\% to 43.6\% on a 15-skill multi-family holdout (n=225), redistributes which InterCode-CTF tasks the model solves, and produces a +33 pp gain in Forensics while reaching 32/100 on InterCode-CTF. These results suggest that Unix competence is separable, trainable, and best evaluated directly rather than folded into programming-through-a-shell.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。