arXiv:2505.12842cs.LGcs.CL2025-05AAAI被引 10

用高斯混合模型检测GUI智能体的异常指令,提升安全性和成功率。

GEM: Gaussian Embedding Modeling for Out-of-Distribution Detection in GUI Agents

  • 基于输入嵌入距离拟合高斯混合模型,捕捉能力边界
  • 平均准确率提升23.70%,测试耗时仅增加6.5%
  • 适用于多种设备与模型,可降低任务失败风险

图形用户界面(GUI)智能体近年来成为人机交互的新范式,能自动执行用户指令操作智能终端。然而,当遇到超出环境约束或自身能力的分布外(OOD)指令时,可能引发任务崩溃甚至安全威胁。现有方法在复杂嵌入空间和动态GUI环境中表现不佳。本文发现GUI智能体的分布内输入语义空间呈现以中心距为特征的聚类模式,据此提出GEM方法:通过在智能体提取的输入嵌入距离上拟合高斯混合模型,建模其能力边界。在涵盖智能手机、电脑和网页浏览器的八个数据集上,该方法相比最优基线平均准确率提升23.70%,训练时间仅增4.9%,测试时间增6.5%。实验还表明,对检测到的OOD样本调用云端模型可使步骤级成功率达9.40%提升。九种不同主干网络的实验验证了方法的泛化能力。代码已开源。

原文摘要 · Abstract (English)

Graphical user interface (GUI) agents have recently emerged as an intriguing paradigm for human-computer interaction, capable of automatically executing user instructions to operate intelligent terminal devices. However, when encountering out-of-distribution (OOD) instructions that violate environmental constraints or exceed the current capabilities of agents, GUI agents may suffer task breakdowns or even pose security threats. Therefore, effective OOD detection for GUI agents is essential. Traditional OOD detection methods perform suboptimally in this domain due to the complex embedding space and evolving GUI environments. In this work, we observe that the in-distribution input semantic space of GUI agents exhibits a clustering pattern with respect to the distance from the centroid. Based on the finding, we propose GEM, a novel method based on fitting a Gaussian mixture model over input embedding distances extracted from the GUI agent that reflect its capability boundary. Evaluated on eight datasets spanning smartphones, computers, and web browsers, our method achieves an average accuracy improvement of 23.70\% over the best-performing baseline while only increasing training time by 4.9\% and testing time by 6.5\%. We also experimentally demonstrate that GEM can improve the step-wise success rate by 9.40\% by requesting assistance from the cloud model when encountering OOD samples. Analysis verifies the generalization ability of our method through experiments on nine different backbones. The codes are available at https://github.com/Wuzheng02/GEM-OODforGUIagents.

GUI智能体异常检测高斯模型安全交互

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。