用API代替界面操作，让智能体与电脑交互更快更准
AXIS: Efficient Human-Agent-Computer Interaction with API-First LLM-Based Agents
- 优先调用API而非逐个点击界面，减少冗余操作
- 在Word中任务耗时降低65%-70%，认知负担减少38%-53%
- 适合开发自动化工具或构建智能代理系统的人
多模态大语言模型使基于LLM的智能体能够直接与应用界面交互，提升复杂任务表现。但这类智能体常因大量串行界面操作导致延迟高、可靠性差。为此，我们提出AXIS框架，通过应用接口（API）优先策略替代界面操作，同时支持通过自动探索应用来创建和扩展API。在Microsoft Word上的实验表明，AXIS将任务完成时间缩短65%-70%，认知负荷降低38%-53%，准确率仍保持在97%-98%（与人类相当）。本工作推动了人-智能体-计算机交互（HACI）新范式，并为应用开发者提供基于LLM的新型界面设计原则，助力实现以智能体为中心的操作系统（Agent OS）。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) have enabled LLM-based agents to directly interact with application user interfaces (UIs), enhancing agents' performance in complex tasks. However, these agents often suffer from high latency and low reliability due to the extensive sequential UI interactions. To address this issue, we propose AXIS, a novel LLM-based agents framework that prioritize actions through application programming interfaces (APIs) over UI actions. This framework also facilitates the creation and expansion of APIs through automated exploration of applications. Our experiments on Microsoft Word demonstrate that AXIS reduces task completion time by 65%-70% and cognitive workload by 38%-53%, while maintaining accuracy of 97%-98% compared to humans. Our work contributes to a new human-agent-computer interaction (HACI) framework and explores a fresh UI design principle for application providers to turn applications into agents in the era of LLMs, paving the way towards an agent-centric operating system (Agent OS).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。