让AI助手学会自我判断,该自己干还是找人帮忙。
OS-Kairos: Adaptive Interaction for MLLM-Powered GUI Agents
- 通过多轮交互标注信心分数,动态评估每步操作的可靠性
- 在复杂场景下任务成功率提升24.59%至87.29%
- 适合需要可靠人机协作的自动化界面应用
由多模态大语言模型驱动的自主图形界面(GUI)代理展现出巨大潜力。然而,一个关键却未被充分探索的问题依然存在:过度执行,即代理在缺乏动作置信度评估的情况下完全自主执行任务,从而损害人机协作的适应性。这在指令模糊、意外中断或环境劫持等复杂场景中带来重大风险。为此,我们提出OS-Kairos,一种具备自适应交互能力的GUI代理,可在每一步交互中预测置信度,并高效决定是否自主执行或寻求人类干预。该系统通过两大机制实现:(i) 协作探查,为每步交互标注置信度分数;(ii) 基于置信度的交互,利用这些分数实现自适应决策。实验结果表明,OS-Kairos在自建的复杂场景数据集及现有基准(AITZ和Meta-GUI)上均显著优于现有模型,任务成功率提升24.59%~87.29%。该方法促进了有效、通用、可扩展且高效的现实世界GUI人机协作。代码与数据集已开源:https://github.com/Wuzheng02/OS-Kairos。
原文摘要 · Abstract (English)
Autonomous graphical user interface (GUI) agents powered by multimodal large language models have shown great promise. However, a critical yet underexplored issue persists: over-execution, where the agent executes tasks in a fully autonomous way, without adequate assessment of its action confidence to compromise an adaptive human-agent collaboration. This poses substantial risks in complex scenarios, such as those involving ambiguous user instructions, unexpected interruptions, and environmental hijacks. To address the issue, we introduce OS-Kairos, an adaptive GUI agent capable of predicting confidence levels at each interaction step and efficiently deciding whether to act autonomously or seek human intervention. OS-Kairos is developed through two key mechanisms: (i) collaborative probing that annotates confidence scores at each interaction step; (ii) confidence-driven interaction that leverages these confidence scores to elicit the ability of adaptive interaction. Experimental results show that OS-Kairos substantially outperforms existing models on our curated dataset featuring complex scenarios, as well as on established benchmarks such as AITZ and Meta-GUI, with 24.59\%$\sim$87.29\% improvements in task success rate. OS-Kairos facilitates an adaptive human-agent collaboration, prioritizing effectiveness, generality, scalability, and efficiency for real-world GUI interaction. The dataset and codes are available at https://github.com/Wuzheng02/OS-Kairos.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。