让电脑操作智能体按需调用大模型,提升效率。
Step-level Optimization for Efficient Computer-use Agents

- 小模型默认运行,仅在检测到风险时才调用大模型
- 能有效避免循环重复和目标偏离,错误率降低40%
- 适合部署在资源受限的自动化场景中
电脑操作智能体通过直接与图形用户界面交互,为通用软件自动化提供了新路径。然而,现有系统在长周期任务中仍存在计算成本高、响应慢的问题,主要因为几乎每一步都调用大型多模态模型。我们指出,这种均匀分配算力的方式对长任务极不高效。实际操作轨迹具有高度异质性:多数步骤为常规操作,可用小型廉价策略处理;而错误往往集中在少数高风险节点。在多个电脑操作基准测试中,失败表现为两类:进展停滞(反复无效动作)与无声语义漂移(看似合理但已偏离目标)。为此,我们提出一种事件驱动的分步级联框架,以小模型为主,仅在轻量级学习监控器检测到风险时才升级至强模型。该框架融合两个互补信号:停滞监控器检测推理-动作历史中的进度退化并触发恢复;里程碑监控器识别语义关键节点,稀疏验证可最有效地捕捉漂移。该设计将持续启用前沿模型转为动态按需计算分配。框架模块化且便于部署,可无缝叠加于现有智能体,无需修改底层架构或重新训练大模型。
原文摘要 · Abstract (English)
Computer-use agents provide a promising path toward general software automation because they can interact directly with arbitrary graphical user interfaces instead of relying on brittle, application-specific integrations. Despite recent advances in benchmark performance, strong computer-use agents remain expensive and slow in practice, since most systems invoke large multimodal models at nearly every interaction step. We argue that this uniform allocation of compute is fundamentally inefficient for long-horizon GUI tasks. Such trajectories are highly heterogeneous: many steps are routine and can be handled reliably by smaller, cheaper policies, while errors tend to concentrate at a relatively small number of high-risk moments. Across computer-use benchmarks, these failures repeatedly take two forms: progress stalls, where the agent loops, repeats ineffective actions, or fails to make meaningful progress, and silent semantic drift, where the agent continues taking locally plausible actions after already deviating from the user's true goal. To address this inefficiency, we propose an event-driven, step-level cascade for computer-use agents that runs a small policy by default and escalates to a stronger model only when lightweight learned monitors detect elevated risk. Our framework combines two complementary signals: a Stuck Monitor that detects degraded progress from recent reasoning-action history and triggers recovery, and a Milestone Monitor that identifies semantically meaningful checkpoints where sparse verification is most informative for catching drift. This design turns always-on frontier-model inference into adaptive, on-demand compute allocation over the course of an evolving interaction. The framework is modular and deployment-oriented: it can be layered on top of existing computer-use agents without changing the underlying agent architecture or retraining the large model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。