用后缀注入加梯度扰动，轻松攻破多模态大模型
Technical Report for ICML 2024 TiFA Workshop MLLM Attack Challenge: Suffix Injection and Projected Gradient Descent Can Easily Fool An MLLM
- 在查询末尾添加错误标签文本作为后缀
- 对图像添加不可察觉的梯度扰动
- 适合研究模型安全与对抗攻击的读者
本技术报告介绍我们针对TiFA工作坊多模态大模型攻击挑战的最优解决方案，采用两种方法：首先将错误标注选项的文本（伪标签）作为后缀附加到原始查询中；其次利用投影梯度下降（PGD）方法对图像添加不可察觉的扰动。结合这两种技术，成功实现了对LLaVA 1.5模型的攻击。
原文摘要 · Abstract (English)
This technical report introduces our top-ranked solution that employs two approaches, \ie suffix injection and projected gradient descent (PGD) , to address the TiFA workshop MLLM attack challenge. Specifically, we first append the text from an incorrectly labeled option (pseudo-labeled) to the original query as a suffix. Using this modified query, our second approach applies the PGD method to add imperceptible perturbations to the image. Combining these two techniques enables successful attacks on the LLaVA 1.5 model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。