对比四种计划表达方式，发现表达形式影响大模型网页代理的完成率。
Does The Way You Plan Matter? An Empirical Study of Planning Representations for LLM Web Agents
- 用四种自然语言计划形式对比效果：步骤列表、叙事、伪代码、清单。
- 不同计划形式下任务成功率差异达18%，伪代码表现最优。
- 适合研究大模型规划能力或开发网页自动化工具的人参考。
尽管近期取得进展，基于大模型的网页代理仍存在探索不足、关键步骤遗漏及对任务约束敏感等问题。已有研究表明，这些问题多源于规划环节的缺陷，但不同自然语言计划表达方式的影响尚未被系统研究。为此，我们提出PlanAhead——一种静态规划-执行框架，用于评估计划表达对代理性能的影响。首先，我们自动将WebArena任务分为3个难度等级，实现无需人工标注的一致性难度划分；随后在高难度任务上系统评估4种计划表达形式（顺序子目标、叙事、伪代码、清单）在三种多模态大模型代理（OpenAI、阿里巴巴、Google）中的表现。为应对随机性，我们引入两项新评估指标：达成率（AR）和已解决问题一致性（STC）。结果表明，计划形式与生成计划的大模型均显著影响网页代理的鲁棒性和任务成功率。
原文摘要 · Abstract (English)
Despite recent advances, LLM-based web agents still struggle with limited exploration, omission of critical steps, and sensitivity to task constraints. Prior work suggests that many of these failures stem from weaknesses in planning, yet the impact of alternative natural language plan representation remains unexplored. To address this, we introduce PlanAhead, a static planner-executor framework that evaluates the impact of plan representation in agent performance. We first automatically categorize WebArena tasks into 3 difficulty levels, enabling consistent difficulty grading without human annotation. Then we systematically evaluate 4 different plan representations on the tasks categorized as hard: sequential subgoals, narrative, pseudocode, and checklist; across different families of multimodal LLM powered agents (OpenAI, Alibaba, and Google). To account for stochastic variability, we introduce two novel evaluation metrics: Achievement Rate (AR) and Solved-Task Consistency (STC). Our results show that both, the plan formulation and the underlying LLM generating the plan, significantly influence web-agent robustness and task success.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。