用树结构优化网页智能体的强化学习，减少冗余操作
TGPO: Tree-Guided Preference Optimization for Robust Web Agent Reinforcement Learning
- 用树形结构合并语义相同的状态，避免标签冲突
- 自动生成细粒度奖励，提升成功率并减少多余步骤
- 适合需要高效网页交互的自动化系统开发者
随着大语言模型和视觉语言模型的发展，使用大模型作为网页智能体已成为实现自动化网页交互的关键。然而，基于强化学习训练网页智能体面临信用分配错误、标注成本过高和奖励稀疏等挑战。为此，我们提出树引导偏好优化（TGPO），一种离线强化学习框架。该框架采用树状轨迹表示，将跨轨迹中语义相同的状态合并，以消除标签冲突。其包含一个流程奖励模型，通过子目标进展、冗余检测和动作验证自动生成细粒度奖励。此外，动态加权机制在训练中优先关注高影响决策点。在Online-Mind2Web和自建C-WebShop数据集上的实验表明，TGPO显著优于现有方法，取得更高的成功率并减少冗余步骤。
原文摘要 · Abstract (English)
With the rapid advancement of large language models and vision-language models, employing large models as Web Agents has become essential for automated web interaction. However, training Web Agents with reinforcement learning faces critical challenges including credit assignment misallocation, prohibitively high annotation costs, and reward sparsity. To address these issues, we propose Tree-Guided Preference Optimization (TGPO), an offline reinforcement learning framework that proposes a tree-structured trajectory representation merging semantically identical states across trajectories to eliminate label conflicts. Our framework incorporates a Process Reward Model that automatically generates fine-grained rewards through subgoal progress, redundancy detection, and action verification. Additionally, a dynamic weighting mechanism prioritizes high-impact decision points during training. Experiments on Online-Mind2Web and our self-constructed C-WebShop datasets demonstrate that TGPO significantly outperforms existing methods, achieving higher success rates with fewer redundant steps.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。