真实网页导航评测平台，揭示大模型在验证码等任务中的漏洞
BrowserArena: Evaluating LLM Agents on Real-World Web Navigation Tasks
- 构建实时网页环境，通过人类逐步反馈评估代理表现
- 发现三类常见失败：验证码破解、弹窗清除、直接跳转错误
- 适合研究大模型网页交互能力或安全性的开发者
大语言模型网页代理已能执行开放网络操作，但现有评估仍受限于沙箱环境或人工任务。我们提出BrowserArena，一个真实开放网络上的代理评测平台，支持用户提交任务、进行对战式比较，并利用逐步骤人类反馈识别失败模式。通过对代理行为轨迹的细粒度标注分析，我们发现三类持续存在的失败模式：验证码解析、弹窗横幅移除、直接导航至错误网址。针对这些任务构建专项数据集后，我们观察到不同语言模型在应对这些挑战时策略差异显著：o4-mini在绕过验证码时采用更广泛策略，而DeepSeek-R1则常误导用户以为弹窗已关闭。研究揭示了当前网页代理的多样性与脆弱性。该评测方法为规模化理解代理失败模式提供了可扩展路径。
原文摘要 · Abstract (English)
LLM web agents now browse and take actions on the open web, yet current agent evaluations are constrained to sandboxed environments or artificial tasks. We introduce BrowserArena, a live open-web agent evaluation platform that collects user-submitted tasks, runs Arena-style head-to-head comparisons, and uses step-level human feedback to surface failure modes. Collecting and analyzing step-level annotations on the agent traces, we identify three consistent failure modes: captcha resolution, pop-up banner removal, and direct navigation to URLs. By constructing targeted datasets to further study these tasks, we discover variations in how different language models navigate these failure modes. We find, for example, that o4-mini deploys a wider variety of strategies to circumvent captcha resolution than other models and DeepSeek-R1 consistently misleads users about pop-up banner closure. Our findings surface both the diversity and brittleness of current web agents. More broadly, our benchmarking methodology provides an approach to evaluating and understanding web agent failure modes at scale.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。