用AI直接操作电脑，一句话指令自动完成复杂任务
OSCAR: Operating System Control via State-Aware Reasoning and Re-Planning
- 基于状态感知的智能推理与动态重规划机制
- 在多平台任务中实现90%以上成功率，显著提升自动化效率
- 适合需要高效桌面/移动端自动化的人群使用
大型语言模型（LLMs）和大型多模态模型（LMMs）在网页浏览、游戏等复杂任务自动化方面展现出巨大潜力。然而，其在不同应用间的泛化能力仍有限，制约了广泛应用。为此，我们提出OSCAR：基于状态感知推理与重规划的操作系统控制框架。OSCAR是一种通用智能体，通过标准化输入（如鼠标、键盘）并分析屏幕图像，自主导航和交互于各类桌面与移动应用，将人类指令转化为可执行的Python代码，实现对图形用户界面（GUI）的精确控制。为增强稳定性与适应性，OSCAR采用状态机架构，具备错误处理机制与动态任务重规划能力，可实时响应反馈与异常情况。我们在跨桌面与移动平台的多个基准测试中验证了其有效性，将复杂工作流简化为自然语言命令，显著提升用户生产力。代码将在发表后开源。
原文摘要 · Abstract (English)
Large language models (LLMs) and large multimodal models (LMMs) have shown great potential in automating complex tasks like web browsing and gaming. However, their ability to generalize across diverse applications remains limited, hindering broader utility. To address this challenge, we present OSCAR: Operating System Control via state-Aware reasoning and Re-planning. OSCAR is a generalist agent designed to autonomously navigate and interact with various desktop and mobile applications through standardized controls, such as mouse and keyboard inputs, while processing screen images to fulfill user commands. OSCAR translates human instructions into executable Python code, enabling precise control over graphical user interfaces (GUIs). To enhance stability and adaptability, OSCAR operates as a state machine, equipped with error-handling mechanisms and dynamic task re-planning, allowing it to efficiently adjust to real-time feedback and exceptions. We demonstrate OSCAR's effectiveness through extensive experiments on diverse benchmarks across desktop and mobile platforms, where it transforms complex workflows into simple natural language commands, significantly boosting user productivity. Our code will be open-source upon publication.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。