通过页面布局匹配复用网页技能,减少大模型交互次数。
Beyond Domains: Reusing Web Skills via Transferable Interaction Patterns
- 用页面结构相似性匹配可迁移的交互模式,而非依赖具体元素。
- 在WebArena和Mind2Web上降低8-10%的平均动作数,成功率相当。
- 适合需要高效网页自动化且跨站点复用能力的场景。
大型语言模型网页代理通常作为工具调用者:每一步读取新页面观测并发出一个结构化动作。当每个动作是低级原语时,决策长度迅速增长,导致政策型大模型生成显著增加延迟与成本,在Mind2Web和WebArena等基准上尤为明显。为此,近期系统将重复交互片段封装为网页技能:由成功轨迹或生成程序构建的可调用工具,单次调用可替代多个原语。但现有技能库仍主要基于指令相似性或粗粒度站点元数据触发,导致在未见站点上的技能复用率低,未能充分实现步骤与令牌的缩减。本文提出SkillMigrator,一种通过匹配布局结构而非特定元素引用,在站点间迁移可复用网页技能的智能体。每个生成的技能以可迁移交互模式(TIP)形式存储:技能与其诱导时刻的页面结构草图配对。测试时,SkillMigrator通过布局相似性检索TIP,并在实时页面上锚定其引用。其余部分保持标准架构:带稳定引用的无障碍快照观测,以及对原语和技能调用的固定工具调用。相较于最先进方法,SkillMigrator在匹配成功率下,使WebArena和Mind2Web上成功轨迹的平均大模型动作数减少8-10%。
原文摘要 · Abstract (English)
Large language model (LLM) web agents are usually deployed as tool callers: each turn, the model reads a fresh page observation and emits one structured tool action. When every action is a low-level primitive, horizons grow quickly and so do policy-facing LLM completions, dominating latency and cost on benchmarks such as Mind2Web and WebArena. Recent systems therefore wrap repeated interaction fragments as web skills: callable tools built from successful trajectories or induced programs, so one call can replace several primitives. However, prior skill libraries are still triggered mainly by instruction similarity or coarse site metadata, which yields low skill reuse on held-out sites and leaves much of the potential step and token reduction on the table. We present SkillMigrator, an agent that learns reusable web skills and transfers them across sites by matching layout structure rather than specific element references. Each induced skill is stored as a transferable interaction pattern (TIP): the skill paired with a structural sketch of the snapshot at induction time. At test time, SkillMigrator retrieves TIPs by layout similarity and grounds their references on the live page. The rest of the stack is standard: accessibility-snapshot observations with stable references, and fixed tool calling over primitives plus skill invocations. Compared with the state-of-the-art approaches, SkillMigrator reduces the average LLM-action count on successful trajectories by 8-10% across both WebArena and Mind2Web at matched success rate.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。