首个评估计算机使用代理挖洞能力的框架,揭示其真实漏洞利用率不足12%。
HackWorld: Evaluating Computer-Use Agents on Exploiting Web Application Vulnerabilities
- 构建包含36个真实应用的网页攻防测试环境
- 实测顶尖代理漏洞利用率低于12%,多步攻击规划能力差
- 适合安全研究者与智能代理开发者参考
网页应用是网络攻击的主要目标,因其连接关键服务和敏感数据。传统渗透测试成本高、依赖专家,难以适应不断增长的网页生态。尽管语言模型代理在网络安全中展现潜力,但现代网页应用需要视觉理解、动态内容处理和多步交互能力,这只有计算机使用代理(CUAs)可实现。然而,它们通过图形界面发现并利用漏洞的能力仍缺乏系统评估。我们提出HackWorld,首个针对CUAs通过可视化交互挖掘网页漏洞的评估框架。不同于简化基准,HackWorld涵盖36个真实应用,涉及11种框架和7种语言,包含注入漏洞、认证绕过、不安全输入处理等实际缺陷。采用夺旗(CTF)形式,测试代理在复杂网页界面中识别和利用弱点的能力。对先进CUAs的评估显示:漏洞利用率低于12%,安全意识薄弱,常无法完成多步攻击规划且误用安全工具。结果揭示了当前CUAs在网页安全场景中的局限性,为开发更具备安全感知能力的智能代理指明方向。
原文摘要 · Abstract (English)
Web applications are prime targets for cyberattacks as gateways to critical services and sensitive data. Traditional penetration testing is costly and expertise-intensive, making it difficult to scale with the growing web ecosystem. While language model agents show promise in cybersecurity, modern web applications demand visual understanding, dynamic content handling, and multi-step interactions that only computer-use agents (CUAs) can perform. Yet, their ability to discover and exploit vulnerabilities through graphical interfaces remains largely unexplored. We present HackWorld, the first framework for systematically evaluating CUAs' capabilities to exploit web application vulnerabilities via visual interaction. Unlike sanitized benchmarks, HackWorld includes 36 real-world applications across 11 frameworks and 7 languages, featuring realistic flaws such as injection vulnerabilities, authentication bypasses, and unsafe input handling. Using a Capture-the-Flag (CTF) setup, it tests CUAs' capacity to identify and exploit these weaknesses while navigating complex web interfaces. Evaluation of state-of-the-art CUAs reveals concerning trends: exploitation rates below 12% and low cybersecurity awareness. CUAs often fail at multi-step attack planning and misuse security tools. These results expose the current limitations of CUAs in web security contexts and highlight opportunities for developing more security-aware agents capable of effective vulnerability detection and exploitation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。