首个评估大模型处理恶意网址漏洞的基准，揭示现有模型识别伪装链接能力差。
MalURLBench: A Benchmark Evaluating Agents' Vulnerabilities When Processing Web URLs
- 构建6万+恶意网址攻击实例，覆盖10类真实威胁场景。
- 12个主流大模型在伪装链接检测中平均准确率不足50%。
- 提出轻量级防御模块URLGuard，适合安全研究与模型开发者使用。
基于大语言模型的网络代理在日常应用中日益普及，但在处理恶意网址时存在严重漏洞：若接受伪装后的恶意链接，可能导致用户访问危险网页，对服务方和用户造成重大损害。然而，当前尚无专门针对此新兴威胁的评测基准。为此，我们提出MalURLBench，首个用于评估大模型对恶意网址脆弱性的基准。该基准包含61,845个攻击实例，涵盖10种真实世界场景及7类真实恶意网站。对12个主流大模型的实验表明，现有模型难以识别精心伪装的恶意网址。我们进一步分析影响攻击成功率的关键因素，并提出轻量级防御模块URLGuard。本工作将为提升网络代理安全性提供基础资源。代码已开源：https://github.com/JiangYingEr/MalURLBench。
原文摘要 · Abstract (English)
LLM-based web agents have become increasingly popular for their utility in daily life and work. However, they exhibit critical vulnerabilities when processing malicious URLs: accepting a disguised malicious URL enables subsequent access to unsafe webpages, which can cause severe damage to service providers and users. Despite this risk, no benchmark currently targets this emerging threat. To address this gap, we propose MalURLBench, the first benchmark for evaluating LLMs' vulnerabilities to malicious URLs. MalURLBench contains 61,845 attack instances spanning 10 real-world scenarios and 7 categories of real malicious websites. Experiments with 12 popular LLMs reveal that existing models struggle to detect elaborately disguised malicious URLs. We further identify and analyze key factors that impact attack success rates and propose URLGuard, a lightweight defense module. We believe this work will provide a foundational resource for advancing the security of web agents. Our code is available at https://github.com/JiangYingEr/MalURLBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。