测试智能助手在真实干扰下的稳定性,发现小问题就导致严重失效。
AgentHijack: Benchmarking Computer Use Agent Robustness to Common Environment Corruptions

- 设计9种可配置环境干扰,模拟真实使用场景
- 微小干扰使多模态大模型助手性能大幅下降
- 提出新框架提升感知与环境判断能力,适合做可靠性评估
基于多模态大语言模型的自主计算机使用代理正成为完成复杂数字任务的有力助手。然而,现实执行环境远非理想:弹窗、分辨率变化及竞争应用频繁干扰代理的感知与控制。我们提出AgentHijack,一个用于评估计算机使用代理在常见环境退化下的鲁棒性基准。该基准引入9种可配置的常见干扰,模拟真实不完美场景。我们评估了多种桌面任务中基于MLLM的代理表现,发现即使轻微干扰也会导致性能显著下降,凸显代理的脆弱性并强调鲁棒性评估的重要性。随后,我们提出AgentHijack-Agent框架,集成具备更强定位能力的动作生成器与负责行为总结和环境检查的旁观者模块。大量实验验证其有效性。代码、环境、基线模型与数据已公开于https://AgentHijack.github.io。
原文摘要 · Abstract (English)
Autonomous computer use agents that powered by multimodal large language models (MLLMs) are emerging as capable assistants for completing complex digital workflows. However, real-world execution environments are far from ideal: pop-ups, resolution changes, and competing applications frequently interfere with agent perception and control. We introduce AgentHijack, a benchmark designed to evaluate the robustness of computer-use agents under common corruptions, where the uncertainties in dynamic environment disrupt the execution flow without direct adversarial intent. Specifically, AgentHijack introduces 9 configurable common corruptions to replicate realistic imperfect scenarios. We evaluate a variety of desktop tasks that utilize MLLM-based agents and discover that even minor instances of corruption can result in substantial performance degradation, which emphasizes the fragility of agents and underscores the necessity of robustness evaluation. Afterward, we propose AgentHijack-Agent, a framework that integrates an action generator with enhanced grounding capabilities and an onlooker responsible for behavior summarization and environment checking. Extensive experiments validate its effectiveness. Our code, environment, baseline models and data are publicly available at: https://AgentHijack.github.io.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。