轻量检测网页截图中的恶意指令注入,速度比GPT-4快8倍。
SnapGuard: Lightweight Prompt Injection Detection for Screenshot-Based Web Agents

- 通过视觉稳定性与文本极性反转分析截图,实现轻量检测。
- 在8种攻击下F1达0.75,速度1.81秒,内存无额外开销。
- 适合部署在资源受限的网页自动化系统中。
网页代理已成为自动化复杂网页交互的有效范式,但易受提示注入攻击——恶意指令被嵌入网页内容以诱导非预期行为。这一威胁在基于截图的网页代理中尤为突出,因其处理的是渲染后的视觉页面而非结构化文本,使主流文本防护手段失效。尽管已有跨模态检测方法,但多依赖大型视觉语言模型(VLM),带来显著计算开销。其瓶颈在于现代网页复杂度:VLM需理解整页全局语义,导致推理时间长、显存占用高。本文提出轻量高效的SnapGuard,将提示注入检测重构为对网页截图的多模态表征分析。该方法利用两个互补信号:视觉稳定性指标(识别恶意内容引起的异常平滑梯度分布)与基于对比极性反转恢复的动作导向文本信号。在八种攻击和两种良性场景下的评估显示,SnapGuard F1得分为0.75,优于GPT-4o-prompt(14.50秒),且仅需1.81秒(快8倍),无额外内存开销。
原文摘要 · Abstract (English)
Web agents have emerged as an effective paradigm for automating interactions with complex web environments, yet remain vulnerable to prompt injection attacks that embed malicious instructions into webpage content to induce unintended actions. This threat is further amplified for screenshot-based web agents, which operate on rendered visual webpages rather than structured textual representations, making predominant text-centric defenses ineffective. Although multimodal detection methods have been explored, they often rely on large vision-language models (VLMs), incurring significant computational overhead. The bottleneck lies in the complexity of modern webpages: VLMs must comprehend the global semantics of an entire page, resulting in substantial inference time and GPU memory usage. This raises a critical question: can we detect prompt injection attacks from screenshots in a lightweight manner? In this paper, we observe that injected webpages exhibit distinct characteristics compared to benign ones from both visual and textual perspectives. Building on this insight, we propose SnapGuard, a lightweight yet accurate method that reformulates prompt injection detection as multimodal representation analysis over webpage screenshots. SnapGuard leverages two complementary signals: a visual stability indicator that identifies abnormally smooth gradient distributions induced by malicious content, and action-oriented textual signals recovered via contrast-polarity reversal. Extensive evaluations across eight attacks and two benign settings demonstrate that SnapGuard achieves an F1 score of 0.75, outperforming GPT-4o-prompt while being 8x faster (1.81s vs. 14.50s) and introducing no additional memory overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。