用视觉语言模型从桌面录像中提取用户操作序列，实现可重放的自动化流程。
Sharingan: Extract User Action Sequence from Desktop Recordings
- 直接输入采样帧给视觉语言模型，无需复杂预处理。
- 准确率70%至80%，提取的动作序列可被机器人流程自动化复现。
- 首次将视觉语言模型用于桌面操作序列提取，适合流程自动化研究者。
桌面操作视频为理解用户行为和自动化流程提供了丰富数据源。尽管视觉语言模型（VLMs）在视频分析中取得进展，但其在桌面录像中的用户动作提取仍属空白。本文提出两种基于VLM的新方法：直接帧输入法（DF），将采样帧直接输入VLM；差分帧法（DiffF），结合计算机视觉检测的帧间差异。我们在自建基础数据集和先前工作改编的进阶基准上评估，结果表明DF方法在识别用户动作上达到70%至80%准确率，且提取的动作序列可由机器人流程自动化系统重播。研究发现，虽然VLM具潜力，但引入显式界面变化反而可能降低性能，因此DF更可靠。本工作首次将VLM应用于桌面操作序列提取，贡献新方法、基准与研究洞见。
原文摘要 · Abstract (English)
Video recordings of user activities, particularly desktop recordings, offer a rich source of data for understanding user behaviors and automating processes. However, despite advancements in Vision-Language Models (VLMs) and their increasing use in video analysis, extracting user actions from desktop recordings remains an underexplored area. This paper addresses this gap by proposing two novel VLM-based methods for user action extraction: the Direct Frame-Based Approach (DF), which inputs sampled frames directly into VLMs, and the Differential Frame-Based Approach (DiffF), which incorporates explicit frame differences detected via computer vision techniques. We evaluate these methods using a basic self-curated dataset and an advanced benchmark adapted from prior work. Our results show that the DF approach achieves an accuracy of 70% to 80% in identifying user actions, with the extracted action sequences being re-playable though Robotic Process Automation. We find that while VLMs show potential, incorporating explicit UI changes can degrade performance, making the DF approach more reliable. This work represents the first application of VLMs for extracting user action sequences from desktop recordings, contributing new methods, benchmarks, and insights for future research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。