arXiv:2509.07553cs.CL2025-09被引 14

让系统在不可靠环境主动问人,提升操作可靠性。

VeriOS: Query-Driven Proactive Human-Agent-GUI Interaction for Trustworthy OS Agents

  • 通过提问机制判断何时需人类介入,避免错误执行。
  • 在不可靠场景下成功率提升19.72%,正常场景性能不变。
  • 适合需要高可信度的本地化智能助手应用。

随着多模态大语言模型的快速发展,操作系统代理已具备通过本地图形界面自动化任务的能力。然而,现有大多数操作系统代理基于理想化假设设计,而真实环境常存在不可信因素。为降低过度执行的风险,本文提出一种查询驱动的人机-代理-GUI交互框架,使代理能在不可信场景下主动向人类求助以更可靠地完成任务。基于此框架,我们构建了VeriOS-Agent,采用三阶段学习范式训练,通过监督微调与组相对策略优化实现元知识的解耦与利用。VeriOS-Agent在正常条件下自主执行,而在不可信场景中主动询问人类。实验表明,其平均逐步成功率比最强基线高出19.72%,且不影响正常性能。在不可信场景中表现显著提升,同时保持与可信场景相当的性能。分析显示该代理具备合理性、泛化性与可扩展性。代码、数据集与模型已开源:https://github.com/Wuzheng02/VeriOS。

原文摘要 · Abstract (English)

With the rapid progress of multimodal large language models, operating system (OS) agents become increasingly capable of automating tasks through on-device graphical user interfaces (GUIs). However, most existing OS agents are designed for idealized settings, whereas real-world environments often present untrustworthy conditions. To mitigate risks of over-execution in such scenarios, we propose a query-driven human-agent-GUI interaction framework that enables OS agents to decide when to query humans for more reliable task completion. Built upon this framework, we introduce VeriOS-Agent, a trustworthy OS agent trained with a three-stage learning paradigm that falicitate the decoupling and utilization of meta-knowledge by supervised fine-tuning and group relative policy optimization. Concretely, VeriOS-Agent autonomously executes actions in normal conditions while proactively querying humans in untrustworthy scenarios. Experiments show that VeriOS-Agent improves the average step-wise success rate by 19.72\% in over the strongest baselines, without compromising normal performance. VeriOS-Agent significantly improves performance in untrustworthy scenarios while maintaining comparable performance in trustworthy scenarios. Analysis highlights VeriOS-Agent's rationality, generalizability, and scalability. The codes, datasets and models are available at https://github.com/Wuzheng02/VeriOS.

可信代理人机协作OS智能

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。