提出可识别敏感界面的探索型智能体，保障用户安全接管。
GUI agent: Guided Exploration of User-Sensitive Screens
- 从示范任务出发，系统探索可能触发敏感状态的指令
- 能发现并分类可能导致隐私泄露的操作场景
- 适合开发需安全交互的GUI自动化工具的工程师
大型语言模型代理正被用于开放图形用户界面中的任务自动化，但它们常遇到包含用户敏感信息的屏幕，此时由用户接管任务执行至关重要。现有最先进的LLM代理通常经过微调以完成任务，而不考虑其行为的安全性，这使得实际部署困难且影响可靠性。因此，识别和分类用户敏感状态，并定义相应的敏感查询至关重要。本短论文提出一种探索型代理，从一个示范任务出发，系统地探索查询空间，以识别出若执行将导致用户敏感状态的指令。该方法有助于工程师在关键场景中识别并请求用户接管，提升系统的安全性与可信度。
原文摘要 · Abstract (English)
LLM agents are increasingly being used to automate tasks for users within an open GUI environment. They inevitably encounter screens containing user-sensitive information, for which takeover of task execution by the user is highly desirable or even necessary. State-of-the-art LLM-driven agents are usually fine-tuned to complete tasks regardless of the safety implications of their actions. This makes their real-world deployment difficult and adversely affects the reliability. Therefore, it is crucial to identify and categorize user-sensitive states and define user-sensitive queries. This dataset would be to engineers to recognize and request handover to the user in critical scenarios. This short paper develops an explorer agent that systematically explores the query space starting from one demonstrated task to identify queries that, if executed, would lead to user-sensitive states in a GUI environment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。