构建真实网页子任务评估基准，测试AI代理精准操作界面的能力。
WARC-Bench: Web Archive Based Benchmark for GUI Subtask Executions
- 用网页存档文件实现动态网页沙盒交互，模拟真实场景
- 顶尖模型在438个子任务中最高成功率仅64.8%
- 强化学习结合可验证奖励能有效提升小数据下的性能
训练网络代理在复杂真实网站上导航，需掌握子任务——即对多个用户界面组件的短时交互（如在日期选择器中选正确日期，或滚动容器提取信息）。我们提出WARC-Bench（Web Archive Benchmark），一个包含438个任务的新式网页导航基准，用于评估多模态AI代理在子任务上的表现。WARC-Bench利用Web ARChive文件实现与动态、真实的网页沙盒式交互。实验表明，该基准对现有领先计算机使用模型极具挑战性，最高成功率为64.8%。为提升开源模型在子任务上的表现，我们探索了两种常见训练方法：监督微调（SFT）和基于可验证奖励的强化学习（RLVR）。实验显示，SFT模型在该基准上取得48.8%的成功率；在数据稀缺情况下，基于RLVR对SFT检查点进行训练，可将成绩提升至52.8%，优于许多前沿模型。分析表明，掌握这些子任务对鲁棒的网页规划与导航至关重要，而这一能力在现有基准中未被充分评估。
原文摘要 · Abstract (English)
Training web agents to navigate complex, real-world websites requires them to master $\textit{subtasks}$ - short-horizon interactions on multiple UI components (e.g., choosing the correct date in a date picker, or scrolling in a container to extract information). We introduce WARC-Bench (Web Archive Benchmark), a novel web navigation benchmark featuring 438 tasks designed to evaluate multimodal AI agents on subtasks. WARC-Bench enables sandboxed interactions with dynamic and realistic webpages using Web ARChive files. We show that WARC-Bench is challenging for leading computer-use models, with the highest observed success rate being 64.8%. To improve open source models on subtask, we explore two common training techniques: supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR). Experiments show that SFT models obtain a 48.8% success rate on the benchmark. Training with RLVR over SFT checkpoints, even in data-scarce settings, improves the score to 52.8% on WARC-Bench, outperforming many frontier models. Our analysis concludes that mastering these subtasks is essential for robust web planning and navigation, and is a capability not extensively evaluated by existing benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。