构建首个完整屏幕解析数据集，提升AI理解界面的精度与效率。
ScreenParse: Moving Beyond Sparse Grounding with Complete Screen Parsing Supervision
- 用自动化流水线生成771万张网页截图，标注全部可见UI元素。
- 自研模型ScreenVLM在密集解析任务上表现远超更大模型（PageIoU 0.592 vs. 0.294）。
- 数据集可迁移性强，适配移动端部署和多场景界面理解任务。
现代计算机使用智能体需将屏幕视为结构化状态，识别可见元素的位置、类型及文本内容，才能准确执行指令。然而现有标注数据集仅提供稀疏监督，标签覆盖不足且多样性差，仅标注少量任务相关元素，限制了模型泛化能力；实际部署还需高效以实现低延迟、本地化运行。本文提出ScreenParse，一个大规模完整屏幕解析数据集，包含771万张网页截图（共2100万个元素），对所有可见UI元素进行密集标注（框、55类类型、文本）。该数据集通过Webshot自动化流水线生成，涵盖多样化网址，自动提取标注并利用视觉语言模型进行重标注与质量过滤。基于ScreenParse，我们训练了轻量级316M参数的ScreenVLM模型，采用结构感知损失，重点优化关键结构标记。ScreenVLM在密集解析任务上显著优于更大基础模型（如PageIoU达0.592对比0.294），并在公开基准上展现良好迁移性。此外，在ScreenParse上微调基础模型可持续提升其指令定位性能，表明密集屏幕监督能提供可迁移的界面结构先验知识。
原文摘要 · Abstract (English)
Modern computer-use agents (CUA) must perceive a screen as a structured state, what elements are visible, where they are, and what text they contain, before they can reliably ground instructions and act. Yet, most available grounding datasets provide sparse supervision, with insufficient and low-diversity labels that annotate only a small subset of task-relevant elements per screen, which limits both coverage and generalization; moreover, practical deployment requires efficiency to enable low-latency, on-device use. We introduce ScreenParse, a large-scale dataset for complete screen parsing, with dense annotations of all visible UI elements (boxes, 55-class types, and text) across 771K web screenshots (21M elements). ScreenParse is generated by Webshot, an automated, scalable pipeline that renders diverse urls, extracts annotations and applies VLM-based relabeling and quality filtering. Using ScreenParse, we train ScreenVLM, a compact, 316M-parameter vision language model (VLM) that decodes a compact ScreenTag markup representation with a structure-aware loss that upweights structure-critical tokens. ScreenVLM substantially outperforms much larger foundation VLMs on dense parsing (e.g., 0.592 vs. 0.294 PageIoU on ScreenParse) and shows strong transfer to public benchmarks. Moreover, finetuning foundation VLMs on ScreenParse consistently improves their grounding performance, suggesting that dense screen supervision provides transferable structural priors for UI understanding. Project page: https://saidgurbuz.github.io/screenparse/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。