将操作日志按子目标分组,能显著提升大模型在模糊指令下的任务成功率。
How Should Agents Read Demonstrations? Hierarchical Structure Beats Flat Action Logs
- 将动作日志按语义分组为带标签的子目标,而非平铺直叙
- 在43个模糊描述任务中,通过率从76.7%提升至90.7%(显著提升)
- 适合非程序员使用演示方式训练大模型代理的场景
编程演示(PbD)为大语言模型代理提供了一种以人为本的程序知识构建方式:用户通过展示操作而非编写提示或代码来表达需求,使非程序员也能轻松创建代理。自然产生的演示输出是扁平的动作日志,但如何组织这些日志传给代理仍是一个未决的设计问题,且对计划质量有重要影响。本文提出将记录的动作按标签分组为层次化的子目标,并在85个网页自动化任务上进行受控实验评估。比较了四种共享相同动作序列但结构不同的演示格式。在43个自然语言描述模糊的任务中,层次化分组的演示将通过率从76.7%提升至90.7%(配对置换检验 $p=0.034$;胜败比6:0),而扁平日志仅带来微小且不显著的改善;在42个描述精确的任务中,任何格式均无提升,说明层次结构的优势仅出现在描述留有程序细节歧义时。消融实验表明,子目标分组本身即为主要驱动力:前置条件、后置条件和参数注释未带来可测量收益。该结果为PbD流水线提供了具体设计建议,也适用于所有向大模型代理输入程序上下文的系统:应将动作序列划分为命名的子目标组,而非呈现为扁平步骤列表。
原文摘要 · Abstract (English)
Programming by Demonstration (PbD) offers a human-centered way to author procedural knowledge for LLM agents: users communicate what they want by showing rather than by writing prompts or code, making agent authoring accessible to non-programmers. The natural output of a PbD recording is a flat action log, but how this log is organized before being passed to the agent is an open design question with significant consequences for plan quality. We propose grouping recorded actions into labeled, hierarchical subgoals and evaluate the effect of this organizational structure in a controlled experiment. Across 85 web automation tasks, we compare a zero-shot baseline against four demonstration formats that share identical action sequences but differ in structure. On 43 natural-language tasks with vague descriptions, hierarchically grouped demonstrations improve pass rates from 76.7\% to 90.7\% (paired permutation test $p{=}0.034$; win-loss 6:0), while flat demonstrations show a smaller, non-significant improvement. On 42 tasks with precise descriptions, no format provides any benefit, confirming that the hierarchical advantage arises specifically when descriptions leave procedural details ambiguous. Ablation shows that subgoal grouping alone drives the effect: preconditions, postconditions, and parameter annotations add no measurable benefit. These results offer a concrete design recommendation for PbD pipelines and, more broadly, for any system that feeds procedural context to an LLM agent: segment action sequences into named subgoal groups rather than presenting flat step lists.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。