让计算机代理先操作程序状态,再处理界面,提升长期任务成功率。
StateAct: Program State, before Pixels, for Long-Horizon Computer-Use Agents

- 用代码直接操作程序状态,仅在必要时调用界面交互
- 在OSWorld 2.0上将成功率从54.8%提升至61.6%,成本降9倍
- 通过状态验证防止文件丢失或路径错误,适合复杂自动化任务
计算机使用代理通常通过增强感知能力来改进,例如更优的截图识别与点击选择模型。然而,截图仅为程序状态(如文件、后端数据、DOM)的低质量呈现,不同状态可能生成相同像素,而代码可直接访问和修改状态。StateAct是一种以代码为核心的多智能体框架,主智能体通过代码直接操作程序状态,仅在28个任务中的少量子目标(占总步骤1.1%)时调用专用的GUI子代理进行截图-点击交互。相同的直接状态访问也支持验证:独立的完成门对保存结果进行结构检查,如文件缺失、未保存或写入错误路径。为保持数百步内的跟踪能力,主智能体将子目标交由新智能体处理,自身上下文保持专注。在OSWorld 2.0上,StateAct使Claude Opus 4.8的二元成功率达26.9%(原20.6%),部分成功率达61.6%(原54.8%),每任务成本降低约9倍;纯代码版本(无GUI子代理)仅达45.9%部分成功率,低于基线。总体而言,将动作、验证和记忆建立在状态之上——即状态接地——使主要瓶颈从感知转向推理:失败更多源于思考而非视觉理解。
原文摘要 · Abstract (English)
Computer-use agents are usually improved by strengthening perception: better models for reading a screenshot and choosing where to click. Yet a screenshot is only a lossy rendering of the underlying program state, e.g., the files, application backends, and DOM that hold the task data. Different states can produce the same pixels, while code can inspect and modify that state directly. StateAct is a code-first, multi-agent harness built around this distinction. Its main agent works directly with program state by using code, while a dedicated GUI subagent handles screenshot-and-click interaction on the few subgoals that need it, just 28 of 108 tasks and 1.1% of main-agent steps. The same direct access to program state also supports verification: an independent finish gate double-checks the saved result for structural failures, e.g., output that is missing, unsaved, or written to the wrong path. To stay on track over hundreds of steps, the main agent hands subgoals to fresh subagents, keeping its own context focused. On OSWorld 2.0, StateAct lifts Claude Opus 4.8 from 20.6% to 26.9% on binary success, and from 54.8% to 61.6% on partial success, at ~ 9x lower cost per task than the same model driven by screenshots alone; a code-only variant with no GUI subagent reaches only 45.9% partial, below that screenshot-based baseline's 54.8%. In general, grounding action, verification, and memory in state, what we call state-grounding, shifts the main bottleneck from perception toward reasoning: failures depend more on what the agent thinks than on what it sees.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。