用视觉技能卡片增强界面操作预测,让AI更懂流程上下文。
SkillLens: Visual Skill Cards for Retrieval-Augmented GUI Action Prediction and On-Policy Distillation

- 构建视觉技能卡片,绑定操作流程与可执行条件
- 在多数据集上提升动作预测准确率11.6~12.0点
- 支持离线训练学生模型,无需推理时查卡
使用计算机的智能体虽能感知丰富软件界面,但其决策常缺乏视觉过程记忆:可能识别单个控件,却无法判断当前流程、下一步关键控件或进度验证依据。原始交互轨迹虽含此类信息,但冗长嘈杂;纯文本技能又常忽略使流程生效的视觉状态。我们提出视觉技能卡片(VSCs),一种状态条件下的记忆表征,将可复用流程与适用性线索、视觉证据及验证信号绑定。SkillLens通过轨迹转视觉技能卡片机制构建VSCs,推理时检索相关卡片,并仅展开固定视觉语言模型执行器所需的证据,实现有依据的GUI动作预测。同一表征还支持CardDistill,利用VSC证据作为特权教师信息,训练无需运行时检索卡片的学生模型。在Multimodal-Mind2Web和WebLINX-BrowserGym数据集上,SkillLens使冻结的GPT-5.4-mini执行器步骤成功率达+11.6,整体表现+2.9;CardDistill进一步使学生模型Qwen3-VL-2B的对应指标提升+12.0和+3.2。
原文摘要 · Abstract (English)
Computer-using agents can perceive rich software interfaces, yet their decisions often lack visual procedural memory: they may recognize individual controls without identifying which familiar workflow is active, which control matters next, or what evidence would confirm progress. Raw interaction traces preserve such information but are long and noisy to condition on, whereas text-only skills often omit the visual state that makes a procedure applicable. We introduce Visual Skill Cards (VSCs), a state-conditioned memory representation that binds reusable procedures with applicability cues, visual evidence, and verification signals. SkillLens constructs VSCs from heterogeneous interaction experience through Trace-to-Visual-Skill-Card and, at inference time, retrieves relevant cards and selectively expands only the evidence needed by a fixed visual-language model executor for grounded GUI action prediction. The same representation also supports CardDistill, which uses VSC evidence as privileged teacher context to train a student that acts without runtime card retrieval. Across Multimodal-Mind2Web and WebLINX-BrowserGym, SkillLens improves the frozen GPT-5.4-mini executor by +11.6 points in Step SR and +2.9 points in Overall, respectively; CardDistill further improves the corresponding student-only Qwen3-VL-2B metrics by +12.0 and +3.2 points.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。