评测智能体在长期重复任务中的表现,发现现有模型效果不佳,单次示范可显著提升适应能力。
OS-Marathon: Benchmarking Computer-Use Agents on Vast-Horizon, Repetitive Tasks
- 构建100个长期重复任务的基准OS-Marathon,覆盖5个场景10个领域
- 顶尖智能体在长周期任务中表现差,错误随执行累积放大
- 仅需一次人类示范即可让通用智能体高效适应重复性工作流
长期重复性工作流广泛存在于日常事务中,如处理大量报销单、整理PDF注释为结构化笔记,这类任务对人类而言枯燥且耗时,但非常适合自主智能体。由于其结构化、可复用的特点,可通过系统化学习掌握。然而,当前缺乏专门针对此类任务的评测基准,导致研究进展有限。为此,我们提出OS-Marathon,包含100个跨5个场景、10个领域的长期重复任务。评估发现,当前最先进的计算机使用智能体(CUAs)在此类任务上表现显著不足。进一步实验表明,简单将任务分解为子任务并由多个代理协作,并不能缓解问题:错误在代理间累积和传播。随后我们提出一种低成本个性化策略GraphDemo,仅需一次人类示范即可使通用智能体学会重复性逻辑。大量实验验证了长期重复任务的真实挑战,以及人类示范在此情境下的有效提升作用。
原文摘要 · Abstract (English)
Vast-horizon, repetitive workflows are common in daily routines, e.g., processing expense reports from a stack of receipts, organising a collection of PDF annotations into structured notes, and are tedious for humans, with execution length scaling with the volume of data to process. They are, however, well-suited to autonomous agents, since their structured, recurring sub-workflows follow a logic that can be systematically learned. Yet no existing benchmark evaluates agents specifically on vast-horizon, repetitive capabilities, leaving the problem largely underexplored. To bridge this gap, we establish OS-Marathon, comprising 100 vast-horizon, repetitive tasks across 5 scenarios and 10 domains, on which our evaluation reveals that leading state-of-the-art (SOTA) CUAs struggle substantially. We further find that introducing a task orchestrator to decompose the workflow into per-instance subtasks fails to mitigate the challenge: errors accumulate and propagate across solver agents, indicating that naive decomposition is insufficient for these tasks. We then explore a cost-friendly personalisation strategy, i.e. GraphDemo, that adapts general agents to such tasks from a single human demonstration of the recurring sub-workflow logic. Extensive experiments show both the real-world challenge of vast-horizon, repetitive tasks and the improvement provided by human demonstration in this setting. Project website: https://os-marathon.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。