让语音助手直接操控界面，实现自然语言操作应用。
A Multimodal GUI Architecture for Interfacing with LLM-Based Conversational Assistants
- 通过MCP协议暴露界面导航与语义，连接语音助手与GUI
- 小型开源语音模型性能接近大厂闭源模型，但需高端硬件支持
- 适合开发语音无障碍应用及未来操作系统级智能助手
大型语言模型（LLMs）和实时语音识别的发展使得用户可通过自然语言下达图形用户界面（GUI）操作指令，并直接在界面上接收系统反馈。然而，大多数现有应用未为语音交互设计。本文提出一种具体架构，使GUI能够与基于LLM的语音助手对接。该架构通过模型上下文协议（MCP）公开应用程序的导航图与语义信息。作为MVVM模式的一部分，ViewModel向助手提供当前视图可用工具及从GUI树路由中提取的应用全局工具。该设计实现全语音可访问性，确保语音输入与视觉界面的一致性对齐，并跨模态提供稳定反馈。同时，该架构为未来操作系统级计算机使用代理（CUAs）做好准备，若应用支持MCP，即可被原生调用。针对隐私与数据安全问题，评估了本地部署、开源权重模型在语音驱动多模态界面中的实际效能。结果表明，近期小型开源模型在整体准确率上接近领先闭源模型，但需企业级硬件以保证响应速度。项目演示可在 https://github.com/hansvdam/langbar 获取。
原文摘要 · Abstract (English)
Advances in large language models (LLMs) and real-time speech recognition now make it possible to issue any graphical user interface (GUI) action through natural language and receive the corresponding system response directly through the GUI. Most production applications were never designed with speech in mind. This article provides a concrete architecture that enables GUIs to interface with LLM-based speech-enabled assistants. The architecture makes an application's navigation graph and semantics available through the Model Context Protocol (MCP). The ViewModel, part of the MVVM (Model-View-ViewModel) pattern, exposes the application's capabilities to the assistant by supplying both tools applicable to a currently visible view and application-global tools extracted from the GUI tree router. This architecture facilitates full voice accessibility while ensuring reliable alignment between spoken input and the visual interface, accompanied by consistent feedback across modalities. It future-proofs apps for upcoming OS super assistants that employ computer use agents (CUAs) and natively consume MCP if an application provides it. To address concerns about privacy and data security, the practical effectiveness of locally deployable, open-weight LLMs for speech-enabled multimodal UIs is evaluated. Findings suggest that recent smaller open-weight models approach the performance of leading proprietary models in overall accuracy and require enterprise-grade hardware for fast responsiveness. A demo implementation of the proposed architecture can be found at https://github.com/hansvdam/langbar
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。