为长任务大模型代理构建可控执行环境,提升稳定性与准确性。
LemonHarness Technical Report

- 通过明确工作区边界,约束文件写入等状态操作。
- 在终端基准测试中达84.49%准确率,强模型组合下升至86.52%。
- 适合需要长时间、多步骤执行的自动化任务研究者。
随着大语言模型代理应用于更长的任务,其在多轮迭代中不断修改工作区状态。然而,代理通常仅能观察工具输出和日志片段,而实际的状态变化发生在文件系统中。由于缺乏明确的工作区边界,文件写入、临时文件生成等状态操作可能分散于多个路径,随时间累积导致已修改文件等状态难以追踪。本文提出LemonHarness,一个面向长时程代理的集成执行框架。该框架通过将模型调用、工具执行和规则知识统一在单一受控边界内,明确限定状态改变操作的范围。所有状态变更(如文件写入、依赖安装、临时文件创建)均通过结构化工具接口执行,并将执行反馈作为后续模型决策的观测信息。系统还引入可复用的规则知识库,将重复出现的执行规则和验收标准转化为运行时知识。此外,新增时间感知机制,向模型暴露已用时间和剩余预算,使其可根据时间压力动态调整探索、实现与验证的投入,避免超时。在Terminal-Bench 2.0上,LemonHarness_GPT-5.3-CodeX在445次试验中达到84.49%准确率;搭配更强的GPT-5.5骨干模型后,五项任务平均准确率达86.52%。结果表明,统一运行边界、可调用规则知识与时间感知机制能显著提升长时程代理执行的稳定性。
原文摘要 · Abstract (English)
As large language model (LLM) agents are applied to longer tasks, they increasingly modify workspace state across multiple rounds of iteration. However, agents typically observe only tool outputs and log fragments, while the actual state changes occur in the file system. Without explicit workspace boundaries, state-changing operations such as file writes and temporary artifact generation may scatter changes across paths. Over time, these weakly constrained changes accumulate, making states such as modified files difficult to track. This paper presents LemonHarness, an integrated execution framework for long-horizon agents. LemonHarness establishes an explicit execution boundary by constraining state-changing operations within a clearly defined workspace and bringing model invocation, tool execution, and rule knowledge within a single controlled boundary. State-changing operations, including file writes, dependency installation, and temporary artifact creation, are executed through structured tool interfaces, with execution feedback recorded as observations available to subsequent model decisions. The system also introduces a reusable rule knowledge base, which turns recurring execution rules and acceptance criteria into runtime knowledge. LemonHarness further adds a time-aware execution mechanism that exposes elapsed and remaining budget to the model, so it can rebalance exploration, implementation, and validation effort as time pressure shifts and avoid timeouts from long waits or excessive verification. On Terminal-Bench 2.0, LemonHarness_GPT-5.3-CodeX reached 84.49% accuracy over 445 trials; pairing the same framework with the stronger GPT-5.5 backbone raised the average accuracy to 86.52% across five jobs. The results suggest that a unified runtime boundary, callable rule knowledge, and time-aware execution can improve the stability of long-horizon agent execution.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。