评测长时间监控类AI代理的响应效率与资源消耗。
SentinelBench: A Benchmark for Long-Running Monitoring Agents
- 设计10个模拟网页环境，测试代理持续监控与响应能力。
- 100项任务中，最佳模型反应时间缩短40%，资源消耗降低35%。
- 适合研究长时任务代理、自动化运维与智能助手的开发者。
AI代理正被用于执行需持续数分钟甚至数小时的任务。然而，当前主流的连续动作模式（如反复调用工具、刷新页面）并不适用于此类任务。更优策略是保持持续关注，等待外部事件触发进展后立即响应，避免无效耗能。为此，我们提出SentinelBench，一个开源基准，用于评估随时间演化的监控类任务。该基准包含100个任务，分布在10个模拟网页环境（如邮件、日历、金融、职业社交、娱乐），每个环境提供实时界面并回放脚本化事件序列，要求代理在状态动态变化的页面中导航与推理。SentinelBench衡量任务完成率、响应时间与资源使用，揭示响应速度与成本间的权衡。我们在三个模型和两个浏览器代理框架上报告结果，建立性能基线，展示代理设计对关键指标的显著影响。结果表明，SentinelBench能有效区分不同代理行为的优劣。
原文摘要 · Abstract (English)
AI agents are increasingly asked to carry out work that spans minutes, hours, or longer. Yet the default model of agent behavior is continuous action: issuing tool calls, refreshing pages, searching for alternatives, or otherwise trying to force progress. This is the wrong approach for many long-running tasks, which are better served by a strategy of sustained attention. Instead, agents should monitor an environment, notice when an external event makes progress possible, then respond promptly without wasting resources while waiting. To measure progress on this class of tasks, we introduce SentinelBench, an open-source benchmark for time-evolving monitoring tasks. SentinelBench contains 100 tasks across 10 synthetic web environments, including email, calendars, finance, professional networking, and entertainment. Each environment exposes a live web interface and replays a scripted sequence of events, requiring agents to navigate and reason about web pages whose state shifts underfoot. SentinelBench measures task completion, reaction time, and resource use, exposing the tradeoff between responsiveness and cost. We report results across three models and two browser-agent harnesses, establishing performance baselines for future comparison and demonstrating how agent design choices can dramatically impact key metrics. Together, these results show that SentinelBench distinguishes meaningful differences in agent behavior.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。