用一张截图自动生成macOS完整可访问性树结构，提升AI代理操作桌面能力。
Screen2AX: Vision-Based Approach for Automatic macOS Accessibility Generation
- 基于视觉语言模型与目标检测，从截图中解析出层级化UI元素
- 在112个应用上重建可访问性树，F1达77%，显著提升代理理解力
- 专为macOS设计的评测基准，比原生方案快2.2倍，优于现有系统
桌面可访问性元数据使AI代理能够解析屏幕，并支持依赖屏幕阅读器的用户。然而，由于开发者提供的元数据不完整或缺失，许多应用仍难以访问——我们调查发现，仅有33%的macOS应用具备完整的可访问性支持。尽管近期研究聚焦于特定挑战如UI元素检测或描述，但尚无工作尝试通过复现整个层级结构来捕捉桌面界面的复杂性。为此，我们提出Screen2AX，首个能从单张截图实时生成树状可访问性元数据的框架。该方法利用视觉-语言和目标检测模型，对UI元素进行检测、描述并分层组织，模拟macOS系统级可访问性结构。为应对macOS桌面应用数据稀缺问题，我们构建并公开发布三个数据集，涵盖112个macOS应用，均包含界面元素检测、分组及层级可访问性元数据标注与对应截图。Screen2AX在重建完整可访问性树方面取得77% F1得分。关键的是，这些层级结构显著提升了自主代理对复杂桌面界面的理解与交互能力。我们引入Screen2AX-Task，一个专为评估macOS桌面环境中自主代理任务执行能力而设计的基准。实验表明，Screen2AX相较原生可访问性表示提升2.2倍性能，并在ScreenSpot基准上超越当前最优的OmniParser V2系统。
原文摘要 · Abstract (English)
Desktop accessibility metadata enables AI agents to interpret screens and supports users who depend on tools like screen readers. Yet, many applications remain largely inaccessible due to incomplete or missing metadata provided by developers - our investigation shows that only 33% of applications on macOS offer full accessibility support. While recent work on structured screen representation has primarily addressed specific challenges, such as UI element detection or captioning, none has attempted to capture the full complexity of desktop interfaces by replicating their entire hierarchical structure. To bridge this gap, we introduce Screen2AX, the first framework to automatically create real-time, tree-structured accessibility metadata from a single screenshot. Our method uses vision-language and object detection models to detect, describe, and organize UI elements hierarchically, mirroring macOS's system-level accessibility structure. To tackle the limited availability of data for macOS desktop applications, we compiled and publicly released three datasets encompassing 112 macOS applications, each annotated for UI element detection, grouping, and hierarchical accessibility metadata alongside corresponding screenshots. Screen2AX accurately infers hierarchy trees, achieving a 77% F1 score in reconstructing a complete accessibility tree. Crucially, these hierarchy trees improve the ability of autonomous agents to interpret and interact with complex desktop interfaces. We introduce Screen2AX-Task, a benchmark specifically designed for evaluating autonomous agent task execution in macOS desktop environments. Using this benchmark, we demonstrate that Screen2AX delivers a 2.2x performance improvement over native accessibility representations and surpasses the state-of-the-art OmniParser V2 system on the ScreenSpot benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。