无需接触模型，用实物文字操控大视觉语言模型行为。
Physical Prompt Injection Attacks on Large Vision-Language Models
- 将恶意指令嵌入物理物体，通过视觉触发攻击。
- 在10个主流模型上成功率最高达98%，抗光照距离干扰强。
- 适合关注AI安全与物理世界攻击面的研究者。
大型视觉语言模型（LVLMs）被广泛部署于开放物理环境中的感知与推理系统。现有提示注入攻击通常需访问输入通道或了解用户查询，但在实际部署中难以满足。本文提出首个物理提示注入攻击（PPIA），一种黑盒、无需查询的攻击方式，将恶意文本指令嵌入视觉可识别的物理对象中。PPIA不依赖模型访问、输入数据或内部流程，仅通过视觉观察即可实施。其结合离线选择高辨识度且语义有效的视觉提示，并利用时空注意力指导环境感知定位，确保注入提示既可见又具影响力。我们在模拟和真实场景下对10个前沿LVLM进行评估，涵盖视觉问答、规划与导航任务，攻击成功率最高达98%，且在不同距离、视角和光照条件下表现稳健。代码已开源：https://github.com/2023cghacker/Physical-Prompt-Injection-Attack。
原文摘要 · Abstract (English)
Large Vision-Language Models (LVLMs) are increasingly deployed in real-world intelligent systems for perception and reasoning in open physical environments. While LVLMs are known to be vulnerable to prompt injection attacks, existing methods either require access to input channels or depend on knowledge of user queries, assumptions that rarely hold in practical deployments. We propose the first Physical Prompt Injection Attack (PPIA), a black-box, query-agnostic attack that embeds malicious typographic instructions into physical objects perceivable by the LVLM. PPIA requires no access to the model, its inputs, or internal pipeline, and operates solely through visual observation. It combines offline selection of highly recognizable and semantically effective visual prompts with strategic environment-aware placement guided by spatiotemporal attention, ensuring that the injected prompts are both perceivable and influential on model behavior. We evaluate PPIA across 10 state-of-the-art LVLMs in both simulated and real-world settings on tasks including visual question answering, planning, and navigation, PPIA achieves attack success rates up to 98%, with strong robustness under varying physical conditions such as distance, viewpoint, and illumination. Our code is publicly available at https://github.com/2023cghacker/Physical-Prompt-Injection-Attack.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。