用专家示范提升GUI代理，搞定复杂自动化任务
Instruction Agent: Enhancing Agent with Expert Demonstration
- 从单次示范中提取步骤指令，严格按轨迹执行
- 在OSWorld任务上达60%成功率，超越所有顶尖代理
- 适合需要可靠桌面自动化的真实场景使用
图形用户界面（GUI）代理虽发展迅速，但在涉及新界面元素、长周期操作和个性化流程的复杂任务上仍表现不佳。本文提出Instruction Agent，通过利用专家示范来解决此类任务，实现原本难以完成的工作流。给定一次示范后，该代理可提取分步指令并严格遵循用户意图的执行轨迹，避免执行错误。此外，其验证器与回溯模块能理解每一步操作的结果，并处理意外中断（如弹窗）。实验表明，Instruction Agent在一组OSWorld任务中达成60%的成功率，而所有顶级代理均未能完成。该框架具备实用性与可扩展性，弥合了当前GUI代理与真实世界自动化之间的差距。
原文摘要 · Abstract (English)
Graphical user interface (GUI) agents have advanced rapidly but still struggle with complex tasks involving novel UI elements, long-horizon actions, and personalized trajectories. In this work, we introduce Instruction Agent, a GUI agent that leverages expert demonstrations to solve such tasks, enabling completion of otherwise difficult workflows. Given a single demonstration, the agent extracts step-by-step instructions and executes them by strictly following the trajectory intended by the user, which avoids making mistakes during execution. The agent leverages the verifier and backtracker modules further to improve robustness. Both modules are critical to understand the current outcome from each action and handle unexpected interruptions(such as pop-up windows) during execution. Our experiments show that Instruction Agent achieves a 60% success rate on a set of tasks in OSWorld that all top-ranked agents failed to complete. The Instruction Agent offers a practical and extensible framework, bridging the gap between current GUI agents and reliable real-world GUI task automation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。