arXiv:2604.23781cs.CVcs.SE2026-04被引 15

构建多日多轮多模态职场助手评测基准,测试模型在动态环境中的持续协作能力。

ClawMark: A Living-World Benchmark for Multi-Turn, Multi-Day, Multimodal Coworker Agents

  • 设计可演化状态的沙盒环境,支持跨天任务与多模态数据交互。
  • 75.8分最高分仍仅20%任务完全成功,体现端到端流程完成难度。
  • 适合评估长期工作代理的适应性与鲁棒性,推动真实场景落地。

语言模型代理正被用于作为跨多日工作的持久同事。在此类任务中,环境会独立于代理而变化:新邮件到达、日程变动、知识库更新,以及图像、扫描件、音频、视频和电子表格中的证据出现。现有基准无法充分评估此场景,因它们通常局限于单次静态回合且以文本为主。我们提出 ench{},一个围绕多轮多日任务构建的职场代理评测基准,包含状态演化的沙盒服务环境及基于规则的验证机制。当前版本涵盖13种专业场景下的100个任务,运行于五个状态化沙盒服务(文件系统、邮件、日历、知识库、电子表格)之上,通过1537个确定性Python检查器对执行后状态评分,不使用LLM作为裁判。我们对七个前沿代理系统进行基准测试,最强模型获75.8分加权得分,但最佳严格任务成功率仅为20.0%,表明部分进展常见而完整流程完成罕见。回合级分析显示,在首次外部环境更新后性能下降,凸显适应状态变化是关键挑战。我们公开基准、评估工具链与构建流水线,以支持可复现的职场代理评估。

原文摘要 · Abstract (English)

Language-model agents are increasingly used as persistent coworkers that assist users across multiple working days. During such workflows, the surrounding environment may change independently of the agent: new emails arrive, calendar entries shift, knowledge-base records are updated, and evidence appears across images, scanned PDFs, audio, video, and spreadsheets. Existing benchmarks do not adequately evaluate this setting because they typically run within a single static episode and remain largely text-centric. We introduce \bench{}, a benchmark for coworker agents built around multi-turn multi-day tasks, a stateful sandboxed service environment whose state evolves between turns, and rule-based verification. The current release contains 100 tasks across 13 professional scenarios, executed against five stateful sandboxed services (filesystem, email, calendar, knowledge base, spreadsheet) and scored by 1537 deterministic Python checkers over post-execution service state; no LLM-as-judge is invoked during scoring. We benchmark seven frontier agent systems. The strongest model reaches 75.8 weighted score, but the best strict Task Success is only 20.0\%, indicating that partial progress is common while complete end-to-end workflow completion remains rare. Turn-level analysis shows that performance drops after the first exogenous environment update, highlighting adaptation to changing state as a key open challenge. We release the benchmark, evaluation harness, and construction pipeline to support reproducible coworker-agent evaluation.

多日任务职场代理动态环境评测基准

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。