文字型视觉提示可诱导跨模态模型生成错误内容，存在安全风险。
Exploring Typographic Visual Prompts Injection Threats in Cross-Modality Generation Models
- 用文字作为视觉提示注入图像，干扰模型输出
- 不同语义的提示均能引发模型误判，影响广泛
- 首次系统评估该威胁，适合关注AI安全的研究者
当前跨模态生成模型在各类生成任务中表现出色。由于视觉输入在现实场景中普遍存在且信息丰富，跨视觉任务（包括视觉-语言感知VLP和图像到图像I2I）受到广泛关注。大型视觉语言模型（LVLMs）和I2I生成模型（GMs）分别用于处理VLP和I2I任务。已有研究发现，在输入图像中嵌入文字型视觉提示，会显著诱导LVLMs和I2I GMs生成与这些文字语义一致的干扰输出。此外，作为更复杂的形式，视觉提示也被证实对多种跨视觉应用构成安全威胁。然而，视觉提示攻击的具体特征仍不明确。本文提出“文字型视觉提示注入数据集”，全面评估多种开源与闭源LVLMs及I2I GMs在不同目标语义视觉提示下的安全风险，深化对典型视觉提示注入（TVPI）威胁的理解。
原文摘要 · Abstract (English)
Current Cross-Modality Generation Models (GMs) demonstrate remarkable capabilities in various generative tasks. Given the ubiquity and information richness of vision modality inputs in real-world scenarios, Cross-Vision tasks, encompassing Vision-Language Perception (VLP) and Image-to-Image (I2I), have attracted significant attention. Large Vision Language Models (LVLMs) and I2I Generation Models (GMs) are employed to handle VLP and I2I tasks, respectively. Previous research indicates that printing typographic words into input images significantly induces LVLMs and I2I GMs to produce disruptive outputs that are semantically aligned with those words. Additionally, visual prompts, as a more sophisticated form of typography, are also revealed to pose security risks to various applications of cross-vision tasks. However, the specific characteristics of the threats posed by visual prompts remain underexplored. In this paper, to comprehensively investigate the performance impact induced by Typographic Visual Prompt Injection (TVPI) in various LVLMs and I2I GMs, we propose the Typographic Visual Prompts Injection Dataset and thoroughly evaluate the TVPI security risks on various open-source and closed-source LVLMs and I2I GMs under visual prompts with different target semantics, deepening the understanding of TVPI threats.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。