通过图像损坏差异,检测大模型是否训练过某张图片。
Image Corruption-Inspired Membership Inference Attacks against Large Vision-Language Models
- 利用图像损坏后嵌入相似度差异区分训练集与非训练集图片。
- 白盒和黑盒场景下均有效,最高攻击准确率达92.3%。
- 无需模型结构知识,适合隐私泄露风险评估场景。
大型视觉语言模型(LVLM)在众多下游任务中表现优异,但其训练数据规模庞大,若包含敏感信息则存在隐私泄露风险。因此,检测某张图像是否用于训练至关重要。现有研究已探索针对LVLM的成员推理攻击,涵盖图像-文本对及单模态内容。本文聚焦于检测目标图像是否被用于训练目标LVLM,提出基于图像损坏启发的成员推理攻击(ICIMIA)。该方法利用LVLM对成员图像与非成员图像在图像损坏下的不同敏感性。在白盒设置下,通过获取图像视觉分支的嵌入,基于原图与其损坏版本的嵌入相似度进行攻击;在更实际的黑盒场景下,仅能通过输入图像与文本指令查询模型,利用输出文本嵌入的相似度实现攻击。在现有数据集上的实验验证了两种设定下方法的有效性。
原文摘要 · Abstract (English)
Large vision-language models (LVLMs) have demonstrated outstanding performance in many downstream tasks. However, LVLMs are trained on large-scale datasets, which can pose privacy risks if training images contain sensitive information. Therefore, it is important to detect whether an image is used to train the LVLM. Recent studies have investigated membership inference attacks (MIAs) against LVLMs, including detecting image-text pairs and single-modality content. In this work, we focus on detecting whether a target image is used to train the target LVLM. We design simple yet effective Image Corruption-Inspired Membership Inference Attacks (ICIMIA) against LVLMs, which are inspired by LVLM's different sensitivity to image corruption for member and non-member images. We first perform an MIA method under the white-box setting, where we can obtain the embeddings of the image through the vision part of the target LVLM. The attacks are based on the embedding similarity between the image and its corrupted version. We further explore a more practical scenario where we have no knowledge about target LVLMs and we can only query the target LVLMs with an image and a textual instruction. We then conduct the attack by utilizing the output text embeddings' similarity. Experiments on existing datasets validate the effectiveness of our proposed methods under those two different settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。