让电脑操作智能体看视频学技能,执行时实时选最相关的操作片段。
Learning from Online Videos at Inference Time for Computer-Use Agents
- 从在线视频中提取结构化操作轨迹,按需动态选择引导
- 在两个基准上超越基线模型,尤其在复杂流程任务中提升显著
- 适合需要学习特定软件操作的自动化工具开发者
计算机使用智能体虽能操作电脑并自动化重复任务,但在涉及特定应用、平台或多步骤流程的领域仍远不及人类。人类可通过观看视频教程,搜索、略读并有选择地模仿匹配当前子目标的短片段来弥补差距。本文研究如何使智能体在推理时有效从在线视频中学习。提出一个框架:检索并过滤教程视频,将其转化为结构化示范轨迹,并在执行过程中动态选择轨迹作为上下文引导。利用视觉语言模型(VLM)推断用户界面动作,将视频分割为短动作序列,并为每段分配文本目标。推理时,采用两阶段选择机制,在每一步动态挑选最具帮助的单一轨迹加入上下文,聚焦于对下一步决策最有价值的局部指导。在两个广泛使用的基准上的实验表明,该框架持续优于强基线模型及仅使用文本教程或转录文本的变体。分析强调了轨迹分割、选择、动作过滤和视觉信息的重要性,表明大量在线视频可系统性提炼为可操作的引导,从而在推理时提升计算机使用智能体的表现。代码已公开于 https://github.com/UCSB-NLP-Chang/video_demo。
原文摘要 · Abstract (English)
Computer-use agents can operate computers and automate laborious tasks, but despite recent rapid progress, they still lag behind human users, especially when tasks require domain-specific procedural knowledge about particular applications, platforms, and multi-step workflows. Humans can bridge this gap by watching video tutorials: we search, skim, and selectively imitate short segments that match our current subgoal. In this paper, we study how to enable computer-use agents to learn from online videos at inference time effectively. We propose a framework that retrieves and filters tutorial videos, converts them into structured demonstration trajectories, and dynamically selects trajectories as in-context guidance during execution. Particularly, using a VLM, we infer UI actions, segment videos into short subsequences of actions, and assign each subsequence a textual objective. At inference time, a two-stage selection mechanism dynamically chooses a single trajectory to add in context at each step, focusing the agent on the most helpful local guidance for its next decision. Experiments on two widely used benchmarks show that our framework consistently outperforms strong base agents and variants that use only textual tutorials or transcripts. Analyses highlight the importance of trajectory segmentation and selection, action filtering, and visual information, suggesting that abundant online videos can be systematically distilled into actionable guidance that improves computer-use agents at inference time. Our code is available at https://github.com/UCSB-NLP-Chang/video_demo.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。