用MCP统一各平台无障碍接口,让AI读屏更智能一致
MCP-Driven Accessibility Tree Standardization for AI-Powered Screen Reader Agents
- 通过MCP协议构建跨平台无障碍树传输层
- 实现多系统下语义信息一致的界面感知
- 适合开发通用型辅助AI交互系统的研究者
基于大语言模型的GUI交互代理通常依赖原始截图或平台专属无障碍API来感知界面状态。前者缺乏屏幕阅读器所需的语义角色与关系,后者如Windows UI Automation、macOS Accessibility、Android AccessibilityService和Web ARIA则需为每平台单独集成。本文提出一种架构,利用模型上下文协议(MCP)作为异构无障碍框架与基于LLM的辅助代理之间的统一传输与模式层。MCP无障碍服务器以平台无关方式暴露符合ARIA的角色、标签、状态及可聚焦元素层级结构,实现跨操作系统与应用的一致交互。该框架还引入MCP资源模型,用于在会话间持久化用户无障碍偏好。针对三个研究问题进行分析:无障碍树表示的协议可扩展性、无障碍树与截图感知间的延迟与语义保真度权衡,以及通过MCP资源支持持久化无障碍配置。本文不提供实证实现,而是基于对无障碍API、GUI代理架构与MCP规范的比较分析,提出概念性框架。分析表明,标准化的MCP无障碍层可在降低平台集成复杂度的同时保留可访问AI代理所需的语义信息,为未来实现与评估奠定基础。
原文摘要 · Abstract (English)
Large language model (LLM) agents that interact with graphical user interfaces increasingly rely on either raw screenshots or platform-specific accessibility application programming interfaces (APIs) to perceive interface state. Both approaches have limitations for assistive applications: screenshot-based perception lacks the semantic roles and relationships required by screen readers, while platform-specific APIs such as Windows UI Automation, macOS Accessibility, Android AccessibilityService, and web ARIA require separate integrations for each platform. This paper proposes an architecture that uses the Model Context Protocol (MCP) as a unified transport and schema layer between heterogeneous accessibility frameworks and LLM-based assistive agents. An MCP accessibility server exposes ARIA-aligned roles, labels, states, and focusable-element hierarchies through a platform-independent representation, enabling consistent interaction across operating systems and applications. The framework also introduces an MCP resource model for persisting user accessibility preferences across sessions. The architecture is analyzed with respect to three research questions: protocol extensibility for accessibility-tree representation, latency and semantic fidelity trade-offs between accessibility trees and screenshot-based perception, and support for persistent accessibility profiles through MCP resources. Rather than presenting an empirical implementation, this work contributes a conceptual framework supported by comparative analysis of accessibility APIs, GUI agent architectures, and the MCP specification. The analysis suggests that a standardized MCP accessibility layer can reduce platform-specific integration complexity while preserving the semantic information required for accessible AI agents, providing a foundation for future implementation and evaluation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。