让大模型用操作系统更高效，只需一句指令完成复杂操作。
From Imperative to Declarative: Towards LLM-friendly OS Interfaces for Boosted Computer-Use Agents
- 用声明式接口抽象图形界面，让大模型专注任务规划
- 在Office套件上任务成功率提升67%，交互步骤减少43.5%
- 超61%的成功任务仅需一次大模型调用，效率显著提升
由大语言模型驱动的计算机使用代理（CUAs）在自动化计算机任务方面展现出巨大潜力，但受限于面向人类设计的图形用户界面（GUI），难以高效执行。现有GUI迫使大模型将高层目标分解为冗长且易错的细粒度操作序列，导致成功率低且调用次数过多。本文提出声明式模型接口（DMI），将现有GUI转化为三种声明式原语：访问、状态与观测，构建专为大模型代理优化的操作系统接口。核心思想是策略与机制分离：大模型专注于高层语义规划（策略），而DMI负责底层导航与交互（机制）。DMI无需修改应用源码或依赖API。我们在Windows系统上的Microsoft Office套件（Word、PowerPoint、Excel）中评估DMI，将其集成到主流基于GUI的代理基线后，任务成功率提升67%，交互步骤减少43.5%。值得注意的是，超过61%的成功任务仅需一次大模型调用即可完成。
原文摘要 · Abstract (English)
Computer-use agents (CUAs) powered by large language models (LLMs) have emerged as a promising approach to automating computer tasks, yet they struggle with the existing human-oriented OS interfaces - graphical user interfaces (GUIs). GUIs force LLMs to decompose high-level goals into lengthy, error-prone sequences of fine-grained actions, resulting in low success rates and an excessive number of LLM calls. We propose Declarative Model Interface (DMI), an abstraction that transforms existing GUIs into three declarative primitives: access, state, and observation, thereby providing novel OS interfaces tailored for LLM agents. Our key idea is policy-mechanism separation: LLMs focus on high-level semantic planning (policy) while DMI handles low-level navigation and interaction (mechanism). DMI does not require modifying the application source code or relying on application programming interfaces (APIs). We evaluate DMI with Microsoft Office Suite (Word, PowerPoint, Excel) on Windows. Integrating DMI into a leading GUI-based agent baseline improves task success rates by 67% and reduces interaction steps by 43.5%. Notably, DMI completes over 61% of successful tasks with a single LLM call.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。