arXiv:2609.02885cs.AIcs.LG2026-09

让网页代理的世界模型更会区分不同操作后果,提升任务成功率。

Discriminative World Models for Web Agents

论文配图:Discriminative World Models for Web Agents
图 1 · 摘自论文原文
  • 用预测状态匹配训练世界模型,使其能区分真实状态与替代动作结果。
  • 在测试中显著提升网页任务成功率,优于传统监督训练方法。
  • 适合需要精准动作评估的自动化网页代理研究者使用。

近期网页代理通过世界模型在测试时进行动作选择:采样候选动作,预测其产生的网页状态,并由排序模型或过程奖励模型(PRM)评分。这些世界模型通常采用监督式下一状态预测训练,生成如HTML或AXTree快照等固定表示。然而,该目标与下游排序模型不一致——后者依赖预测状态在候选动作间具有可区分性以准确打分。为此,本文提出预测状态匹配训练目标:要求预测表示能区分真实结果状态与其它动作导致的状态。我们在基于WebArena Go-Browse轨迹构建的分支型网页代理数据集上训练模型,每个决策点包含多个替代动作及其对应状态。在预留的预测状态匹配基准测试中,本方法优于传统监督训练的世界模型。进一步实验表明,在WebPRMBench上,该方法改进了基于PRM的动作排序效果,优于仅动作级PRM及融合监督下一状态世界模型的PRM。在WebArena-Lite上,使用该世界模型进行测试时动作选择,显著提升了端到端任务成功率。

原文摘要 · Abstract (English)

Recent web agents use world models for test-time action selection by sampling candidate actions, predicting the resulting web states, and ranking them with a ranker model or a Process Reward Model (PRM). These world models are typically trained via supervised next-state prediction to generate fixed representations like HTML or AXTree snapshots. However, this objective is misaligned with the downstream ranker, which relies on predicted states being discriminative across candidates to accurately score them. To address this, we introduce predicted-state matching, a training objective where the predicted representation must distinguish the true resulting state from those reached by alternative actions. We train these models using a branching web-agent dataset derived from WebArena Go-Browse trajectories, where every decision point contains multiple alternative actions and their resulting states. Experiments on our held-out predicted-state matching benchmark show that our approach outperforms world models trained with supervised next-state prediction. We further show that our approach improves PRM-style action ranking on WebPRMBench compared with action-only PRMs and PRMs augmented with supervised-next-state world models. Finally, on WebArena-Lite, using our world model for test-time action selection improves end-to-end task success. Our project page is available at: https://dhruvpendharkar.github.io/dwm/.

网页代理世界模型动作排序强化学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。